2024-03-29T10:28:11.872 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:28:11.872 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/orig.config.yaml 2024-03-29T10:28:11.908 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:28:12.005 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622646' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBET4uWJ1dRCY8i6GpiJNqX+aKDBuXu7e+8lWovKubi62FhnlCX+Wyv2xAdWjc9TNzP2I0g6FSbsqwBbCUYvKKv4= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA0TujK83NAzIr2yYFWZMtxTPs5Qz3YLnVMBZnma4ddeq0wqf85ZCyNNfWNT6kIfQwZeb7neKme59t5iHy5j8Oo= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:28:12.006 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:28:12.007 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:28:12.007 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:28:12.045 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:28:12.074 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:28:12.074 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:28:12.074 INFO:teuthology.packaging:ref: None 2024-03-29T10:28:12.075 INFO:teuthology.packaging:tag: None 2024-03-29T10:28:12.075 INFO:teuthology.packaging:branch: main 2024-03-29T10:28:12.075 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:28:12.075 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:28:12.465 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:28:12.466 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:28:12.585 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:28:12.585 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:28:12.606 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:28:12.621 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:28:12.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:28:12.656 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:23:03.018362', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBET4uWJ1dRCY8i6GpiJNqX+aKDBuXu7e+8lWovKubi62FhnlCX+Wyv2xAdWjc9TNzP2I0g6FSbsqwBbCUYvKKv4='} 2024-03-29T10:28:12.678 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:23:03.019772', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA0TujK83NAzIr2yYFWZMtxTPs5Qz3YLnVMBZnma4ddeq0wqf85ZCyNNfWNT6kIfQwZeb7neKme59t5iHy5j8Oo='} 2024-03-29T10:28:12.678 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:28:12.686 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:28:12.686 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:28:12.686 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:28:12.766 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9862a240d0>, signals=[15]) 2024-03-29T10:28:12.766 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:28:12.774 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:28:12.774 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:28:12.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:28:12.869 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:28:12.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:28:12.958 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:28:12.966 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-03-29T10:28:12.985 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-03-29T10:28:12.985 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:28:13.045 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:28:13.046 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:28:13.046 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:28:13.046 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:28:13.046 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-03-29T10:28:13.068 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-03-29T10:28:13.086 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-03-29T10:28:13.086 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:28:13.144 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:28:13.145 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:28:13.145 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:28:13.145 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:28:13.145 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:28:13.145 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-03-29T10:28:13.159 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:28:13.169 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:28:13.175 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:28:13.175 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:28:13.177 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:28:13.201 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:28:13.207 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:28:13.207 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:28:13.236 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:28:13.310 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:28:13.364 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:28:13.380 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:28:13.381 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:28:13.381 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:28:13.381 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:28:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:28:13.396 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-29T10:28:13.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:28:13.397 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-29T10:28:13.454 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:28:13.454 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-361.el9.x86_64 2024-03-29T10:28:13.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-03-29T10:28:13.455 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:28:13.455 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-361.el9.x86_64 2024-03-29T10:28:13.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-03-29T10:28:15.668 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:28:15.677 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:28:17.045 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T10:28:17.102 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-29T10:28:18.010 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T10:28:18.218 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:28:20.356 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-29T10:28:24.708 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.1 MB 00:00 2024-03-29T10:28:25.868 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-03-29T10:28:27.172 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-29T10:28:29.118 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:28:31.127 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-29T10:28:32.727 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T10:28:34.132 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T10:28:35.156 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-29T10:28:36.448 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:28:36.448 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:28:36.545 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:28:36.556 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:28:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:28:36.557 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:28:36.557 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:28:36.557 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-03-29T10:28:36.557 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:28:36.557 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 76 M 2024-03-29T10:28:36.557 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 123 M 2024-03-29T10:28:36.558 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:28:36.701 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:28:37.351 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-29T10:28:37.893 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-03-29T10:28:38.362 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T10:28:38.365 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:28:38.365 INFO:teuthology.orchestra.run.smithi094.stdout:Total 42 MB/s | 76 MB 00:01 2024-03-29T10:28:38.366 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:28:38.724 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:28:38.724 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:28:39.545 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:28:39.546 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:28:40.070 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T10:28:41.861 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:28:42.310 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:28:42.310 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:28:42.419 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:28:42.428 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:28:42.428 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:28:42.428 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:28:42.428 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:28:42.428 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:28:42.428 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-03-29T10:28:42.429 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:28:42.430 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 76 M 2024-03-29T10:28:42.430 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 123 M 2024-03-29T10:28:42.430 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:28:42.504 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:28:42.596 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T10:28:42.602 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:43.298 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-29T10:28:43.938 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-29T10:28:44.330 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T10:28:44.333 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:28:44.333 INFO:teuthology.orchestra.run.smithi173.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-29T10:28:44.333 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:28:44.730 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:28:44.730 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:28:45.351 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:45.515 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:45.613 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:28:45.614 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:28:47.932 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:28:48.512 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:28:48.605 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:50.245 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:50.298 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:28:51.405 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:28:51.558 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:55.026 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:28:56.299 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:28:56.407 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:01.133 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:24.565 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:24.565 INFO:teuthology.orchestra.run.smithi094.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:29:24.565 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:29:24.617 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:24.986 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:29:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:29:24.987 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:29:25.425 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:29:25.759 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:25.855 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:25.855 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:25.855 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:29:25.856 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:29:25.856 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:29:25.856 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:29:25.856 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi094.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:29:25.856 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:25.936 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:25.937 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:25.937 DEBUG:teuthology.orchestra.run.smithi094:> sudo rpm -qi grub2-tools 2024-03-29T10:29:25.977 INFO:teuthology.orchestra.run.smithi094.stdout:Name : grub2-tools 2024-03-29T10:29:25.977 INFO:teuthology.orchestra.run.smithi094.stdout:Epoch : 1 2024-03-29T10:29:25.977 INFO:teuthology.orchestra.run.smithi094.stdout:Version : 2.06 2024-03-29T10:29:25.977 INFO:teuthology.orchestra.run.smithi094.stdout:Release : 68.el9 2024-03-29T10:29:25.977 INFO:teuthology.orchestra.run.smithi094.stdout:Architecture: x86_64 2024-03-29T10:29:25.977 INFO:teuthology.orchestra.run.smithi094.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Group : Unspecified 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Size : 8264254 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:License : GPLv3+ 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Packager : builder@centos.org 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Vendor : CentOS 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Summary : Support tools for GRUB. 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:Description : 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:29:25.978 INFO:teuthology.orchestra.run.smithi094.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:29:25.979 INFO:teuthology.orchestra.run.smithi094.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:29:25.979 INFO:teuthology.orchestra.run.smithi094.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:29:25.979 INFO:teuthology.orchestra.run.smithi094.stdout:hardware devices. 2024-03-29T10:29:25.979 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:29:25.979 INFO:teuthology.orchestra.run.smithi094.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:29:25.980 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:29:25.980 INFO:teuthology.task.kernel:Updating grub on smithi094 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:29:25.980 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:29:26.678 INFO:teuthology.orchestra.run.smithi094.stderr:Generating grub configuration file ... 2024-03-29T10:29:28.080 INFO:teuthology.orchestra.run.smithi094.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:29:28.122 INFO:teuthology.orchestra.run.smithi094.stderr:done 2024-03-29T10:29:28.124 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-29T10:29:28.139 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.Gzj2R85bNR 2024-03-29T10:29:28.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Gzj2R85bNR 2024-03-29T10:29:28.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.Gzj2R85bNR 2024-03-29T10:29:28.350 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.Gzj2R85bNR is 6KB 2024-03-29T10:29:28.398 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.Gzj2R85bNR 2024-03-29T10:29:28.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:29:28.479 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:29:28.480 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:29:28.480 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:29:28.480 INFO:teuthology.orchestra.run.smithi094.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:29:28.481 DEBUG:teuthology.orchestra.run.smithi094:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:29:28.614 DEBUG:teuthology.orchestra.run.smithi094:> sudo shutdown -r now 2024-03-29T10:29:30.833 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:30.833 INFO:teuthology.orchestra.run.smithi173.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:29:30.834 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:29:30.877 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:29:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:29:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:29:31.686 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:29:32.011 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:32.117 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:32.118 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:32.118 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:29:32.118 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:29:32.118 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:29:32.118 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:29:32.118 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:29:32.118 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:29:32.199 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:29:32.199 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:29:32.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-03-29T10:29:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:29:32.251 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:29:32.252 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:29:32.252 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:29:32.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:29:32.848 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-03-29T10:29:34.209 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:29:34.243 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-03-29T10:29:34.246 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-29T10:29:34.261 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.RyzqTDxuj7 2024-03-29T10:29:34.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RyzqTDxuj7 2024-03-29T10:29:34.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.RyzqTDxuj7 2024-03-29T10:29:34.474 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.RyzqTDxuj7 is 6KB 2024-03-29T10:29:34.523 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.RyzqTDxuj7 2024-03-29T10:29:34.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:29:34.604 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:29:34.604 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:29:34.604 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:29:34.604 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:29:34.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:29:34.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-03-29T10:29:58.640 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:29:58.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T10:29:58.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:04.763 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:30:04.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T10:30:04.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:30:58.696 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:31:04.773 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:31:07.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T10:31:07.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:09.103 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-29T10:31:09.618 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T10:31:09.618 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:31:09.621 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-03-29T10:31:09.635 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:31:09.636 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:31:09.636 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:31:09.636 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:31:10.637 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:31:10.637 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:31:10.759 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-03-29T10:31:10.810 DEBUG:teuthology.parallel:result is None 2024-03-29T10:31:13.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T10:31:13.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:14.385 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-29T10:31:14.857 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T10:31:14.858 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:31:14.858 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-29T10:31:14.873 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:31:14.873 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:31:14.873 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:31:14.873 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:31:15.874 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:31:15.874 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:31:15.936 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-03-29T10:31:15.975 DEBUG:teuthology.parallel:result is None 2024-03-29T10:31:15.976 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:31:15.983 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:31:15.983 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:31:15.986 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:31:16.011 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:31:16.017 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:31:16.028 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:31:16.028 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:31:16.057 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:31:16.096 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:31:16.103 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:31:16.103 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:31:16.138 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:31:16.178 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:31:16.184 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:31:16.196 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:31:16.197 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:31:16.199 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:31:16.207 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:31:16.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:31:16.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:31:16.287 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:31:16.296 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:31:16.296 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:31:16.334 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:31:16.356 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:31:16.444 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:31:16.507 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:31:16.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:31:16.640 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:31:16.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:31:16.709 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:31:16.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:31:16.810 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:31:16.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:31:16.936 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:31:16.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:31:17.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-03-29T10:31:17.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-03-29T10:31:17.057 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:31:17.096 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:31:17.424 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:31:17.433 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:31:17.434 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:31:17.445 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:31:17.453 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-03-29T10:31:17.508 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-03-29T10:31:17.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-03-29T10:31:17.570 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-03-29T10:31:17.571 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:31:17.572 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-03-29T10:31:17.594 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-03-29T10:31:17.594 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-03-29T10:31:17.620 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-03-29T10:31:17.620 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:31:17.620 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:31:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:31:17.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:31:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:31:17.689 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:31:17.689 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:31:17.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:31:17.837 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:31:17.847 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:31:17.863 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:31:17.864 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi094.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:35:36.788 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-03-29T10:35:36.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T10:35:36.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:36.880 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-03-29T10:35:36.960 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-03-29T10:35:36.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T10:35:36.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:35:37.050 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-29T10:35:37.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-29T10:35:37.125 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:35:37.136 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:35:37.136 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:35:37.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:35:37.139 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:35:37.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:35:37.169 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:35:37.185 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:35:37.198 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:35:37.215 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:35:37.215 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-03-29T10:35:37.228 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-03-29T10:35:37.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-03-29T10:35:37.245 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:35:37.258 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-03-29T10:35:37.261 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:35:37.274 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:35:37.290 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:35:37.311 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:35:37.314 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:35:37.314 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-29T10:35:37.314 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.314 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.314 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.314 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.344 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:35:37.346 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:35:37.346 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-29T10:35:37.346 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.347 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.347 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.347 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:35:37.348 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:35:37.357 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:35:37.357 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:35:37.357 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:35:37.357 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:35:37.363 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:35:37.363 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:35:37.364 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:35:37.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:35:37.366 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:35:37.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:35:37.621 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:35:37.621 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:35:37.625 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:35:37.625 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:35:37.743 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:35:37.744 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:35:37.744 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:35:37.771 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:35:37.772 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:35:37.839 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:35:37.857 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:35:37.857 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:35:37.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:35:37.885 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:35:37.885 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:35:37.919 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:35:37.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:35:37.987 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-03-29T10:35:37.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-03-29T10:35:38.033 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:35:38.099 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-03-29T10:35:38.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-03-29T10:35:38.226 INFO:teuthology.orchestra.run.smithi094.stdout:74 files removed 2024-03-29T10:35:38.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-03-29T10:35:38.344 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-03-29T10:35:38.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-03-29T10:35:38.932 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 214 kB/s | 82 kB 00:00 2024-03-29T10:35:39.001 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 248 kB/s | 82 kB 00:00 2024-03-29T10:35:39.291 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-29T10:35:39.360 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-29T10:35:39.600 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-03-29T10:35:39.752 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-29T10:35:40.285 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-29T10:35:40.472 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-29T10:35:41.906 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-29T10:35:41.994 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-29T10:35:43.638 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-29T10:35:45.560 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-29T10:35:46.339 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T10:35:50.990 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-03-29T10:35:52.708 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-29T10:35:54.402 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T10:35:59.418 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T10:36:02.042 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:36:02.058 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:02.058 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:36:02.058 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:36:02.059 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:36:02.060 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout:Install 30 Packages 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-03-29T10:36:02.061 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:02.062 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 58 M 2024-03-29T10:36:02.062 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:36:03.169 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 5.6 MB/s | 5.3 MB 00:00 2024-03-29T10:36:03.219 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 505 kB/s | 25 kB 00:00 2024-03-29T10:36:03.353 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-29T10:36:03.528 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.4 MB/s | 9.7 MB 00:01 2024-03-29T10:36:03.553 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.4 MB/s | 498 kB 00:00 2024-03-29T10:36:03.595 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T10:36:03.646 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T10:36:03.704 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T10:36:03.780 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.2 MB/s | 320 kB 00:00 2024-03-29T10:36:03.872 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 15 MB/s | 5.1 MB 00:00 2024-03-29T10:36:03.897 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.5 MB/s | 298 kB 00:00 2024-03-29T10:36:03.922 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:36:04.048 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-03-29T10:36:04.073 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-29T10:36:04.115 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 663 kB/s | 44 kB 00:00 2024-03-29T10:36:04.199 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 661 kB 00:00 2024-03-29T10:36:04.224 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-03-29T10:36:04.249 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-29T10:36:04.291 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T10:36:04.317 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 523 kB/s | 35 kB 00:00 2024-03-29T10:36:04.342 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-03-29T10:36:04.584 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-29T10:36:04.643 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 448 kB/s | 26 kB 00:00 2024-03-29T10:36:04.743 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-03-29T10:36:04.843 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-29T10:36:04.910 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-03-29T10:36:04.977 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T10:36:05.019 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-29T10:36:05.203 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.7 MB/s | 20 MB 00:02 2024-03-29T10:36:05.254 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.7 MB/s | 1.6 MB 00:00 2024-03-29T10:36:05.470 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-03-29T10:36:05.537 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.5 MB/s | 3.4 MB 00:00 2024-03-29T10:36:05.542 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:05.542 INFO:teuthology.orchestra.run.smithi173.stdout:Total 17 MB/s | 58 MB 00:03 2024-03-29T10:36:05.542 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:36:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:36:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:36:06.049 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:36:06.049 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:36:07.503 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:36:07.585 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:36:07.617 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:36:07.642 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:36:07.666 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:07.683 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:36:07.880 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:36:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:36:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:08.362 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:08.451 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:36:08.479 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:36:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:36:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:36:08.615 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:36:08.692 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:36:08.740 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:36:08.804 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:36:08.839 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:36:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:36:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:36:09.023 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:36:09.055 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:36:09.084 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:36:09.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:36:09.259 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 854 kB/s | 19 MB 00:22 2024-03-29T10:36:09.439 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:36:09.472 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:36:09.561 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:36:09.593 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:36:09.633 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:36:10.084 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:36:10.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:36:10.188 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:36:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:36:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:36:12.385 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:12.433 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:12.989 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:13.028 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:36:13.114 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:36:13.115 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:36:13.115 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:36:13.115 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:13.154 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:13.201 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T10:36:15.052 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-03-29T10:36:17.089 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:36:22.029 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T10:36:23.022 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:23.022 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-03-29T10:36:23.023 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-03-29T10:36:23.023 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-03-29T10:36:23.023 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-03-29T10:36:23.023 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-03-29T10:36:23.023 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-03-29T10:36:23.023 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-03-29T10:36:23.023 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:23.889 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:36:23.913 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:36:23.914 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:23.914 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:36:23.914 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:36:23.914 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:36:23.914 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:23.915 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:36:24.008 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:36:24.008 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:24.722 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:36:24.738 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:36:24.739 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:36:24.740 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:36:24.741 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout:Install 30 Packages 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:24.742 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 58 M 2024-03-29T10:36:24.743 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:36:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:36:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:36:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:36:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:36:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:36:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:36:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:36:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:36:26.030 INFO:teuthology.orchestra.run.smithi094.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.7 MB/s | 5.3 MB 00:01 2024-03-29T10:36:26.079 INFO:teuthology.orchestra.run.smithi094.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 25 kB 00:00 2024-03-29T10:36:26.213 INFO:teuthology.orchestra.run.smithi094.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-29T10:36:26.330 INFO:teuthology.orchestra.run.smithi094.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.2 MB/s | 498 kB 00:00 2024-03-29T10:36:26.476 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:26.476 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:26.476 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.477 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.478 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:26.479 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:36:26.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-03-29T10:36:26.947 INFO:teuthology.orchestra.run.smithi094.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.3 MB/s | 5.1 MB 00:00 2024-03-29T10:36:26.997 INFO:teuthology.orchestra.run.smithi094.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 906 kB/s | 45 kB 00:00 2024-03-29T10:36:27.081 INFO:teuthology.orchestra.run.smithi094.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T10:36:27.148 INFO:teuthology.orchestra.run.smithi094.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.4 MB/s | 162 kB 00:00 2024-03-29T10:36:27.265 INFO:teuthology.orchestra.run.smithi094.stdout:(9/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.1 MB/s | 9.7 MB 00:02 2024-03-29T10:36:27.290 INFO:teuthology.orchestra.run.smithi094.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.2 MB/s | 320 kB 00:00 2024-03-29T10:36:27.340 INFO:teuthology.orchestra.run.smithi094.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:36:27.366 INFO:teuthology.orchestra.run.smithi094.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 298 kB 00:00 2024-03-29T10:36:27.369 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:36:27.483 INFO:teuthology.orchestra.run.smithi094.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-03-29T10:36:27.508 INFO:teuthology.orchestra.run.smithi094.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-29T10:36:27.525 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:36:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-29T10:36:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:27.528 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 48 M 2024-03-29T10:36:27.528 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 204 M 2024-03-29T10:36:27.528 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:36:27.533 INFO:teuthology.orchestra.run.smithi094.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-29T10:36:27.583 INFO:teuthology.orchestra.run.smithi094.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T10:36:27.609 INFO:teuthology.orchestra.run.smithi094.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T10:36:27.634 INFO:teuthology.orchestra.run.smithi094.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-03-29T10:36:27.659 INFO:teuthology.orchestra.run.smithi094.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T10:36:27.684 INFO:teuthology.orchestra.run.smithi094.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 701 kB/s | 35 kB 00:00 2024-03-29T10:36:27.710 INFO:teuthology.orchestra.run.smithi094.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T10:36:27.948 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T10:36:28.015 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 300 kB 00:00 2024-03-29T10:36:28.048 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:36:28.073 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 586 kB/s | 163 kB 00:00 2024-03-29T10:36:28.199 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:36:28.210 INFO:teuthology.orchestra.run.smithi094.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.0 MB/s | 20 MB 00:03 2024-03-29T10:36:28.310 INFO:teuthology.orchestra.run.smithi094.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 494 kB/s | 309 kB 00:00 2024-03-29T10:36:28.394 INFO:teuthology.orchestra.run.smithi094.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-03-29T10:36:28.477 INFO:teuthology.orchestra.run.smithi094.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-29T10:36:28.503 INFO:teuthology.orchestra.run.smithi094.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-03-29T10:36:28.620 INFO:teuthology.orchestra.run.smithi094.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-29T10:36:28.811 INFO:teuthology.orchestra.run.smithi094.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-29T10:36:28.870 INFO:teuthology.orchestra.run.smithi094.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-29T10:36:29.220 INFO:teuthology.orchestra.run.smithi094.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 8.3 MB/s | 3.4 MB 00:00 2024-03-29T10:36:29.287 INFO:teuthology.orchestra.run.smithi094.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.5 MB/s | 3.1 MB 00:00 2024-03-29T10:36:29.554 INFO:teuthology.orchestra.run.smithi094.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-03-29T10:36:29.561 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:29.561 INFO:teuthology.orchestra.run.smithi094.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-29T10:36:29.562 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:36:29.599 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:36:29.599 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:36:30.091 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:36:30.091 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:36:31.374 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 13 MB/s | 47 MB 00:03 2024-03-29T10:36:31.377 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:31.377 INFO:teuthology.orchestra.run.smithi173.stdout:Total 13 MB/s | 48 MB 00:03 2024-03-29T10:36:31.391 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:36:31.412 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:36:31.412 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:36:31.554 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:36:31.632 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:36:31.659 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:36:31.684 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:36:31.716 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:31.732 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:31.793 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:36:31.794 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:36:31.924 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:36:31.987 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:36:32.314 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:36:32.367 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:36:32.394 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:32.433 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:36:32.468 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:32.473 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:36:32.523 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:36:32.555 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:36:32.574 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:36:32.587 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:36:32.602 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:36:32.641 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:36:32.670 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:36:32.713 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:36:32.780 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:36:32.805 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:36:32.852 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:36:32.896 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:36:33.002 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:36:33.030 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:36:33.086 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:36:33.121 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:36:33.149 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:36:33.182 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:36:33.506 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:36:33.536 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:36:33.635 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:36:33.666 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:36:33.711 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:36:34.218 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:36:34.247 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:36:34.310 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:36:34.378 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:36:34.416 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:36:36.602 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:36.645 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:37.082 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:36:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:36:37.185 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:36:37.224 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:36:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:36:37.315 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:36:37.315 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:36:37.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:37.350 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:36:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:36:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:36:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:36:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:36:39.661 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:36:40.040 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:36:40.040 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:40.040 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:36:40.040 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:40.040 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:36:40.040 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:40.040 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:36:40.041 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:36:40.041 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:36:40.041 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:40.041 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:36:40.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-03-29T10:36:40.906 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:36:41.059 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:36:41.061 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-29T10:36:41.061 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:36:41.061 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-29T10:36:41.061 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:36:41.062 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:36:41.063 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout:Install 36 Packages 2024-03-29T10:36:41.064 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:41.065 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 30 M 2024-03-29T10:36:41.065 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 107 M 2024-03-29T10:36:41.065 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:36:41.485 INFO:teuthology.orchestra.run.smithi173.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-03-29T10:36:41.627 INFO:teuthology.orchestra.run.smithi173.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-03-29T10:36:41.694 INFO:teuthology.orchestra.run.smithi173.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.8 MB/s | 2.1 MB 00:00 2024-03-29T10:36:41.745 INFO:teuthology.orchestra.run.smithi173.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.8 MB/s | 247 kB 00:00 2024-03-29T10:36:41.862 INFO:teuthology.orchestra.run.smithi173.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T10:36:41.904 INFO:teuthology.orchestra.run.smithi173.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:36:41.937 INFO:teuthology.orchestra.run.smithi173.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-29T10:36:41.971 INFO:teuthology.orchestra.run.smithi173.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:36:42.012 INFO:teuthology.orchestra.run.smithi173.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 932 kB/s | 39 kB 00:00 2024-03-29T10:36:42.063 INFO:teuthology.orchestra.run.smithi173.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:36:42.188 INFO:teuthology.orchestra.run.smithi173.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 9.2 MB/s | 1.1 MB 00:00 2024-03-29T10:36:42.230 INFO:teuthology.orchestra.run.smithi173.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-29T10:36:42.272 INFO:teuthology.orchestra.run.smithi173.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T10:36:42.314 INFO:teuthology.orchestra.run.smithi173.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T10:36:42.531 INFO:teuthology.orchestra.run.smithi173.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 576 kB/s | 125 kB 00:00 2024-03-29T10:36:42.740 INFO:teuthology.orchestra.run.smithi173.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T10:36:42.757 INFO:teuthology.orchestra.run.smithi173.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-29T10:36:42.857 INFO:teuthology.orchestra.run.smithi173.stdout:(18/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.4 MB/s | 4.7 MB 00:01 2024-03-29T10:36:42.874 INFO:teuthology.orchestra.run.smithi173.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-03-29T10:36:42.924 INFO:teuthology.orchestra.run.smithi173.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 821 kB/s | 151 kB 00:00 2024-03-29T10:36:42.983 INFO:teuthology.orchestra.run.smithi173.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T10:36:43.017 INFO:teuthology.orchestra.run.smithi173.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-29T10:36:43.042 INFO:teuthology.orchestra.run.smithi173.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-03-29T10:36:43.075 INFO:teuthology.orchestra.run.smithi173.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-29T10:36:43.101 INFO:teuthology.orchestra.run.smithi173.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 331 kB/s | 19 kB 00:00 2024-03-29T10:36:43.126 INFO:teuthology.orchestra.run.smithi173.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-03-29T10:36:43.168 INFO:teuthology.orchestra.run.smithi173.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 694 kB/s | 46 kB 00:00 2024-03-29T10:36:43.193 INFO:teuthology.orchestra.run.smithi173.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T10:36:43.235 INFO:teuthology.orchestra.run.smithi173.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-03-29T10:36:43.260 INFO:teuthology.orchestra.run.smithi173.stdout:(30/36): python3-jaraco-collections-3.0.0-8.el9 95 kB/s | 23 kB 00:00 2024-03-29T10:36:43.294 INFO:teuthology.orchestra.run.smithi173.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T10:36:43.361 INFO:teuthology.orchestra.run.smithi173.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-29T10:36:43.386 INFO:teuthology.orchestra.run.smithi173.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-29T10:36:43.437 INFO:teuthology.orchestra.run.smithi173.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-03-29T10:36:43.487 INFO:teuthology.orchestra.run.smithi173.stdout:(35/36): python3-tempora-5.0.0-2.el9.noarch.rpm 158 kB/s | 36 kB 00:00 2024-03-29T10:36:43.679 INFO:teuthology.orchestra.run.smithi173.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.3 MB/s | 427 kB 00:00 2024-03-29T10:36:43.682 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:43.682 INFO:teuthology.orchestra.run.smithi173.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T10:36:43.707 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:36:43.751 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:36:43.751 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:36:44.164 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:36:44.165 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:36:45.483 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:36:45.575 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:36:45.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:36:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:36:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:36:45.782 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:36:45.815 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:36:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:36:45.978 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:36:46.026 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:36:46.105 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:36:46.160 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:36:46.198 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:36:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:36:46.303 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:36:46.489 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:36:46.544 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:36:46.575 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:36:46.617 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:36:46.648 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:36:46.723 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:36:46.758 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:36:46.791 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:36:46.840 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:36:46.884 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:36:46.926 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:36:47.016 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:36:47.056 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:36:47.130 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:36:47.268 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-03-29T10:36:47.289 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-03-29T10:36:47.290 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:47.392 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:36:47.557 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:36:47.587 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:36:47.613 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:36:47.613 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:47.613 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:36:47.613 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:36:47.613 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:36:47.613 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:48.197 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:36:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:36:48.222 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:48.222 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:36:48.222 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:36:48.222 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:36:48.222 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:48.224 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:36:48.305 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:36:48.305 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:49.080 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:36:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:36:49.107 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:49.107 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:36:49.107 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:36:49.107 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:36:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:49.448 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:36:49.449 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:36:49.450 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:36:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:36:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:36:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:36:49.451 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:36:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:36:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:36:49.579 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:49.579 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:36:49.579 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:36:49.579 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:36:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:49.781 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:36:49.804 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:36:49.804 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:49.804 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:36:49.804 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:36:49.804 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:36:49.804 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:49.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.715 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:50.717 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:36:51.074 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-03-29T10:36:51.638 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:36:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:36:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:36:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:36:51.691 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:36:51.692 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:36:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:36:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:36:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:36:51.693 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:36:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:36:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:36:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:36:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:36:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:36:51.694 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:36:51.794 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:36:51.795 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 48 M 2024-03-29T10:36:51.796 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 204 M 2024-03-29T10:36:51.797 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:36:52.069 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:36:52.119 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:36:52.145 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T10:36:52.279 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 441 kB/s | 163 kB 00:00 2024-03-29T10:36:52.337 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 975 kB/s | 187 kB 00:00 2024-03-29T10:36:52.749 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:36:52.749 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:52.749 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:36:52.749 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:36:52.750 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:36:52.751 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:52.752 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:36:53.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-03-29T10:36:53.625 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:36:53.723 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:36:53.777 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:36:53.778 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:36:53.778 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:36:53.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-03-29T10:36:54.407 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:36:54.557 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:54.558 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 747 k 2024-03-29T10:36:54.559 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 754 k 2024-03-29T10:36:54.559 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:36:55.364 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 928 kB/s | 747 kB 00:00 2024-03-29T10:36:55.365 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:55.365 INFO:teuthology.orchestra.run.smithi173.stdout:Total 926 kB/s | 747 kB 00:00 2024-03-29T10:36:55.365 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:36:55.370 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:36:55.370 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:36:55.376 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:36:55.376 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:36:55.448 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:36:55.562 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 13 MB/s | 47 MB 00:03 2024-03-29T10:36:55.565 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:55.565 INFO:teuthology.orchestra.run.smithi094.stdout:Total 13 MB/s | 48 MB 00:03 2024-03-29T10:36:55.579 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:36:55.600 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:36:55.600 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:36:55.931 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:55.979 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:36:55.980 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:36:56.034 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:56.434 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:56.553 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:36:56.737 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:36:56.737 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:56.737 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:36:56.737 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:36:56.737 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:56.737 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:36:56.738 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:36:56.781 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:36:56.822 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:36:56.863 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:36:56.892 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:36:56.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:36:57.501 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:36:57.655 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:36:57.656 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:57.656 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T10:36:57.656 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:57.656 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:36:57.656 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:36:57.656 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:57.657 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:36:57.657 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:36:57.657 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T10:36:57.657 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:57.657 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 145 k 2024-03-29T10:36:57.657 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 436 k 2024-03-29T10:36:57.657 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:36:57.887 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 634 kB/s | 145 kB 00:00 2024-03-29T10:36:57.887 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:57.887 INFO:teuthology.orchestra.run.smithi173.stdout:Total 630 kB/s | 145 kB 00:00 2024-03-29T10:36:57.887 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:36:57.894 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:36:57.894 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:36:57.948 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:36:57.948 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:36:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:36:58.145 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:36:58.170 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:36:58.170 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:36:58.170 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:36:58.170 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:59.168 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:36:59.168 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:36:59.169 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:59.169 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:36:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:36:59.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-03-29T10:37:00.010 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:00.108 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:00.160 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:00.161 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:00.161 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:00.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:37:00.784 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:00.935 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:00.936 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-29T10:37:00.936 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:37:00.936 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-29T10:37:00.936 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:00.936 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:37:00.936 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:37:00.936 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-29T10:37:00.937 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-29T10:37:00.938 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:00.938 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.9 M 2024-03-29T10:37:00.938 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-29T10:37:00.938 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:01.357 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 167 kB/s | 31 kB 00:00 2024-03-29T10:37:01.457 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 53 kB/s | 15 kB 00:00 2024-03-29T10:37:01.468 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:37:01.516 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T10:37:01.541 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:37:01.549 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T10:37:01.725 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-03-29T10:37:01.850 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.6 MB/s | 1.8 MB 00:00 2024-03-29T10:37:01.967 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-29T10:37:02.017 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T10:37:02.084 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 463 kB/s | 31 kB 00:00 2024-03-29T10:37:02.134 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 869 kB/s | 144 kB 00:00 2024-03-29T10:37:02.167 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T10:37:02.385 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.8 MB/s | 2.0 MB 00:00 2024-03-29T10:37:02.390 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:02.390 INFO:teuthology.orchestra.run.smithi173.stdout:Total 4.7 MB/s | 6.9 MB 00:01 2024-03-29T10:37:02.408 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:02.425 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:02.425 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:02.601 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:02.601 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:03.210 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:37:03.254 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:37:03.301 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:37:03.339 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:37:03.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:37:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:37:03.684 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:37:03.735 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:37:03.763 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:37:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:37:04.038 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:37:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:37:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:37:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:37:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:37:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:37:04.041 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:37:04.079 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:37:04.436 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:04.437 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:04.725 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:37:04.932 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:37:04.933 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:37:04.933 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:37:04.933 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:37:05.291 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:05.294 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:37:05.295 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:37:05.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:05.296 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:05.455 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:05.457 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-29T10:37:05.457 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:05.457 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-29T10:37:05.457 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:05.457 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:37:05.457 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:37:05.457 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:37:05.458 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:37:05.459 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:37:05.459 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:37:05.461 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:37:05.461 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:37:05.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout:========================================================================================= 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout:Install 36 Packages 2024-03-29T10:37:05.463 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:05.464 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 30 M 2024-03-29T10:37:05.464 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 107 M 2024-03-29T10:37:05.464 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:05.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:37:06.109 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:06.262 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:06.264 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-29T10:37:06.264 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:37:06.265 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:37:06.266 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:37:06.266 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:06.266 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:06.267 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-29T10:37:06.267 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-29T10:37:06.267 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:06.267 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 37 M 2024-03-29T10:37:06.267 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 209 M 2024-03-29T10:37:06.267 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:06.301 INFO:teuthology.orchestra.run.smithi094.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 43 kB/s | 6.6 kB 00:00 2024-03-29T10:37:06.533 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-29T10:37:06.566 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-29T10:37:06.567 INFO:teuthology.orchestra.run.smithi094.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-03-29T10:37:06.609 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-29T10:37:06.667 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:37:06.934 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-29T10:37:06.935 INFO:teuthology.orchestra.run.smithi094.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 7.4 MB/s | 4.7 MB 00:00 2024-03-29T10:37:06.985 INFO:teuthology.orchestra.run.smithi094.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 2.1 MB 00:00 2024-03-29T10:37:07.001 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-03-29T10:37:07.010 INFO:teuthology.orchestra.run.smithi094.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T10:37:07.035 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-29T10:37:07.218 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-29T10:37:07.219 INFO:teuthology.orchestra.run.smithi094.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 799 kB/s | 187 kB 00:00 2024-03-29T10:37:07.244 INFO:teuthology.orchestra.run.smithi094.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 687 kB/s | 144 kB 00:00 2024-03-29T10:37:07.260 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 195 kB 00:00 2024-03-29T10:37:07.270 INFO:teuthology.orchestra.run.smithi094.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-03-29T10:37:07.303 INFO:teuthology.orchestra.run.smithi094.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 809 kB/s | 47 kB 00:00 2024-03-29T10:37:07.329 INFO:teuthology.orchestra.run.smithi094.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 655 kB/s | 39 kB 00:00 2024-03-29T10:37:07.379 INFO:teuthology.orchestra.run.smithi094.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-29T10:37:07.429 INFO:teuthology.orchestra.run.smithi094.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-29T10:37:07.455 INFO:teuthology.orchestra.run.smithi094.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T10:37:07.477 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-29T10:37:07.480 INFO:teuthology.orchestra.run.smithi094.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:37:07.505 INFO:teuthology.orchestra.run.smithi094.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T10:37:07.589 INFO:teuthology.orchestra.run.smithi094.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-29T10:37:07.619 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 688 kB/s | 245 kB 00:00 2024-03-29T10:37:07.631 INFO:teuthology.orchestra.run.smithi094.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 830 kB/s | 125 kB 00:00 2024-03-29T10:37:07.853 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.1 MB/s | 7.4 MB 00:01 2024-03-29T10:37:07.854 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:07.855 INFO:teuthology.orchestra.run.smithi173.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-29T10:37:07.881 INFO:teuthology.orchestra.run.smithi094.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 17 MB 00:01 2024-03-29T10:37:07.895 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:07.906 INFO:teuthology.orchestra.run.smithi094.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 158 kB/s | 43 kB 00:00 2024-03-29T10:37:07.919 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:07.919 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:07.998 INFO:teuthology.orchestra.run.smithi094.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 369 kB/s | 151 kB 00:00 2024-03-29T10:37:08.057 INFO:teuthology.orchestra.run.smithi094.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T10:37:08.082 INFO:teuthology.orchestra.run.smithi094.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-03-29T10:37:08.107 INFO:teuthology.orchestra.run.smithi094.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 760 kB/s | 172 kB 00:00 2024-03-29T10:37:08.124 INFO:teuthology.orchestra.run.smithi094.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-03-29T10:37:08.141 INFO:teuthology.orchestra.run.smithi094.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 392 kB/s | 23 kB 00:00 2024-03-29T10:37:08.175 INFO:teuthology.orchestra.run.smithi094.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-03-29T10:37:08.192 INFO:teuthology.orchestra.run.smithi094.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 292 kB/s | 20 kB 00:00 2024-03-29T10:37:08.209 INFO:teuthology.orchestra.run.smithi094.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 688 kB/s | 46 kB 00:00 2024-03-29T10:37:08.259 INFO:teuthology.orchestra.run.smithi094.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-03-29T10:37:08.276 INFO:teuthology.orchestra.run.smithi094.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-29T10:37:08.289 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:08.289 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:08.335 INFO:teuthology.orchestra.run.smithi094.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T10:37:08.352 INFO:teuthology.orchestra.run.smithi094.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-29T10:37:08.435 INFO:teuthology.orchestra.run.smithi094.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-29T10:37:08.461 INFO:teuthology.orchestra.run.smithi094.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-03-29T10:37:08.486 INFO:teuthology.orchestra.run.smithi094.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T10:37:08.503 INFO:teuthology.orchestra.run.smithi094.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 298 kB/s | 20 kB 00:00 2024-03-29T10:37:08.506 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:08.506 INFO:teuthology.orchestra.run.smithi094.stdout:Total 9.7 MB/s | 30 MB 00:03 2024-03-29T10:37:08.531 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:08.573 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:08.573 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:08.988 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:08.989 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:09.018 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:09.109 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:37:09.148 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:37:09.241 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:37:09.277 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:37:09.453 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:37:09.484 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:37:09.565 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:37:10.268 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:37:10.347 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:10.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:37:10.406 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:37:10.439 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:37:10.467 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:37:10.537 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:37:10.582 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:37:10.615 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:37:10.708 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:37:10.794 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:37:10.834 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:37:10.908 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:37:10.968 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:37:11.016 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:37:11.073 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:37:11.136 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:37:11.322 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:37:11.370 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:37:11.417 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:37:11.452 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:37:11.482 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:37:11.559 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:37:11.592 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:37:11.626 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:37:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:37:11.666 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:37:11.710 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:37:11.753 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:37:11.841 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:37:11.892 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:37:11.973 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:37:12.109 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:37:12.236 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:37:12.400 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:37:12.431 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:37:12.456 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:37:12.456 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:12.456 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:37:12.456 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:37:12.457 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:37:12.457 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:37:12.691 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:37:13.898 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:37:13.925 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:37:13.925 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:13.925 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:37:13.925 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:37:13.925 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:37:13.926 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:14.366 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:37:14.389 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:37:14.389 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:14.389 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:37:14.389 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:37:14.389 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:37:14.389 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:14.600 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:37:14.622 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:37:14.622 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:14.623 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:37:14.623 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:37:14.623 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:37:14.623 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:14.698 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:37:15.450 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:37:15.450 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:37:15.451 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:37:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:37:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:15.905 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:37:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:37:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:37:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:37:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:37:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:37:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:37:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:37:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:37:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:37:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:37:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:15.907 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:16.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:37:16.674 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:37:16.675 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:37:16.676 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:37:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:37:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:37:16.677 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:37:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:37:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:37:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:37:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:37:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:37:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:37:16.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:37:16.877 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:17.028 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:17.029 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:37:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-03-29T10:37:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:17.032 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.0 M 2024-03-29T10:37:17.032 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-29T10:37:17.032 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:17.317 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-03-29T10:37:17.343 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-29T10:37:17.385 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T10:37:17.410 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T10:37:17.435 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:37:17.452 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T10:37:17.469 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 188 kB/s | 52 kB 00:00 2024-03-29T10:37:17.506 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:37:17.506 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:17.507 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:17.507 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:17.507 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:17.507 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:17.507 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:17.507 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:17.507 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:37:17.508 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:37:17.509 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:37:17.510 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:37:17.510 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:37:17.510 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:37:17.510 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:37:17.510 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:17.510 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:17.844 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2024-03-29T10:37:17.861 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 76 kB/s | 32 kB 00:00 2024-03-29T10:37:17.894 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-03-29T10:37:18.061 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 296 kB/s | 59 kB 00:00 2024-03-29T10:37:18.120 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 311 kB/s | 203 kB 00:00 2024-03-29T10:37:18.220 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 564 kB/s | 90 kB 00:00 2024-03-29T10:37:18.421 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-29T10:37:18.425 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:18.425 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-29T10:37:18.426 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:18.439 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:18.439 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:18.460 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:18.532 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:18.533 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:18.558 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:18.609 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:18.610 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:37:18.611 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:18.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-03-29T10:37:19.037 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:37:19.239 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:19.271 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:37:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:37:19.381 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:37:19.388 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:19.389 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:19.390 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T10:37:19.390 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:19.390 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 747 k 2024-03-29T10:37:19.390 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 754 k 2024-03-29T10:37:19.390 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:19.422 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:37:19.524 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:37:19.632 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:37:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:37:19.706 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.3 MB/s | 747 kB 00:00 2024-03-29T10:37:19.706 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:19.707 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.3 MB/s | 747 kB 00:00 2024-03-29T10:37:19.707 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:19.712 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:19.712 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:19.717 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:19.718 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:19.792 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:20.139 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:20.203 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:20.401 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:37:20.454 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:37:20.496 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:37:20.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:37:20.575 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:37:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:20.894 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:20.894 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:20.895 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:21.118 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:37:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:37:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:37:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:37:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:37:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:37:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:37:21.691 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:37:21.695 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:37:21.696 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:37:21.696 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:37:21.696 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:37:21.696 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:37:21.696 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:21.696 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:21.844 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:21.845 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T10:37:21.846 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:21.846 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 145 k 2024-03-29T10:37:21.846 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 436 k 2024-03-29T10:37:21.846 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:21.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:37:22.121 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 529 kB/s | 145 kB 00:00 2024-03-29T10:37:22.121 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:22.122 INFO:teuthology.orchestra.run.smithi094.stdout:Total 526 kB/s | 145 kB 00:00 2024-03-29T10:37:22.122 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:22.130 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:22.130 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:22.182 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:22.182 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:22.325 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:22.397 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:22.421 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:22.421 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:22.422 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:37:22.422 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:22.535 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:22.688 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:37:22.689 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:37:22.690 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:37:22.690 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:22.690 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:22.690 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-29T10:37:22.690 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-29T10:37:22.690 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:22.690 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.8 M 2024-03-29T10:37:22.691 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-03-29T10:37:22.691 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:23.248 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T10:37:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:23.406 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:23.406 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:23.406 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:23.564 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 287 kB/s | 138 kB 00:00 2024-03-29T10:37:23.590 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T10:37:23.648 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 878 kB/s | 51 kB 00:00 2024-03-29T10:37:23.658 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-03-29T10:37:23.790 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 256 kB/s | 58 kB 00:00 2024-03-29T10:37:23.882 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.2 MB/s | 5.8 MB 00:00 2024-03-29T10:37:23.885 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:23.885 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-29T10:37:23.897 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:23.907 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:23.907 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:23.995 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:23.996 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:24.233 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:24.317 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:24.331 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:24.383 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:24.384 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:37:24.384 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:24.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:37:24.477 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:37:24.533 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:37:25.012 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:25.046 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:37:25.128 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:37:25.163 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:37:25.164 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout:====================================================================================== 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-03-29T10:37:25.165 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:25.166 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.9 M 2024-03-29T10:37:25.166 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-03-29T10:37:25.166 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:25.197 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:37:25.221 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:37:25.826 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 73 kB/s | 15 kB 00:00 2024-03-29T10:37:25.876 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 121 kB/s | 31 kB 00:00 2024-03-29T10:37:25.976 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.4 MB/s | 238 kB 00:00 2024-03-29T10:37:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:37:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:37:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:37:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:37:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:37:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:37:26.026 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:26.303 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:26.335 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.5 MB/s | 1.8 MB 00:00 2024-03-29T10:37:26.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-03-29T10:37:26.577 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 81 kB/s | 19 kB 00:00 2024-03-29T10:37:26.636 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 900 kB/s | 548 kB 00:00 2024-03-29T10:37:26.745 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-29T10:37:26.836 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 335 kB/s | 31 kB 00:00 2024-03-29T10:37:26.945 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T10:37:27.094 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:27.178 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-03-29T10:37:27.246 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:27.246 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:27.246 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:37:27.246 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-29T10:37:27.247 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:27.248 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 964 k 2024-03-29T10:37:27.248 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.8 M 2024-03-29T10:37:27.248 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:27.367 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 663 kB/s | 79 kB 00:00 2024-03-29T10:37:27.567 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-03-29T10:37:27.569 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:27.569 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.9 MB/s | 964 kB 00:00 2024-03-29T10:37:27.570 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:27.588 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:27.588 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:27.653 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 515 kB/s | 863 kB 00:01 2024-03-29T10:37:27.658 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:27.658 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-29T10:37:27.661 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:27.661 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:27.678 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:27.696 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:27.696 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:27.851 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:27.884 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:27.884 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:28.011 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:37:28.083 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:37:28.389 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:28.471 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:37:28.506 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:37:28.561 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:37:28.600 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:37:28.702 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:37:28.834 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:37:28.863 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:37:28.864 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:37:28.929 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:37:28.981 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:37:29.008 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:37:29.040 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:37:29.183 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:37:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:29.184 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:37:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:29.184 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:29.284 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:37:29.315 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:37:29.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-volume 2024-03-29T10:37:30.056 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:30.128 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:37:30.128 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:37:30.128 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:37:30.128 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:37:30.128 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:37:30.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:37:30.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:37:30.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:37:30.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:37:30.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:37:30.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:37:30.129 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:37:30.208 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:30.209 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-29T10:37:30.210 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:30.210 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 332 k 2024-03-29T10:37:30.210 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 1.5 M 2024-03-29T10:37:30.210 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:30.332 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 564 kB/s | 69 kB 00:00 2024-03-29T10:37:30.466 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-29T10:37:30.467 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:30.467 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T10:37:30.467 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:30.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:37:30.472 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:30.472 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:30.472 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:37:30.472 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:30.473 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:30.475 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:30.475 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:30.515 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:30.515 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:30.661 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:30.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:37:30.848 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:37:30.902 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:37:30.918 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:37:30.918 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:30.918 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:37:30.918 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:31.308 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:31.459 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:31.460 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:37:31.461 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:37:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:37:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:37:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:31.462 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:31.462 INFO:teuthology.orchestra.run.smithi094.stdout:============================================================================================ 2024-03-29T10:37:31.462 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-03-29T10:37:31.462 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:31.463 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 37 M 2024-03-29T10:37:31.463 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 209 M 2024-03-29T10:37:31.463 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:31.639 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:37:31.712 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 208 kB/s | 32 kB 00:00 2024-03-29T10:37:31.745 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 524 kB/s | 17 kB 00:00 2024-03-29T10:37:31.779 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:37:31.829 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:37:31.989 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:37:31.990 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:31.990 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:31.990 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:31.990 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:37:31.990 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:31.990 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:32.021 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:37:32.097 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T10:37:32.122 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:37:32.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-03-29T10:37:32.322 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-03-29T10:37:32.373 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-29T10:37:32.548 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-29T10:37:32.698 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 6.5 MB/s | 7.4 MB 00:01 2024-03-29T10:37:32.723 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 705 kB/s | 245 kB 00:00 2024-03-29T10:37:32.726 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:32.726 INFO:teuthology.orchestra.run.smithi094.stdout:Total 29 MB/s | 37 MB 00:01 2024-03-29T10:37:32.767 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:32.791 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:32.791 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:32.864 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:33.015 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:33.016 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 127 k 2024-03-29T10:37:33.017 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-03-29T10:37:33.017 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:33.164 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:33.164 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:33.363 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 366 kB/s | 127 kB 00:00 2024-03-29T10:37:33.364 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:33.364 INFO:teuthology.orchestra.run.smithi173.stdout:Total 364 kB/s | 127 kB 00:00 2024-03-29T10:37:33.365 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:33.370 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:33.370 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:33.426 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:33.426 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:33.563 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:33.670 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:33.906 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:33.988 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:37:34.019 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:37:34.113 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:37:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:34.156 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:37:34.333 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:37:34.355 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:37:34.419 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:34.419 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:34.419 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:34.419 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:34.419 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:34.419 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:34.435 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:37:34.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-03-29T10:37:35.111 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:37:35.244 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:37:35.255 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:35.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-03-29T10:37:36.049 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:36.201 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 k 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 140 k 2024-03-29T10:37:36.202 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:36.367 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 190 kB/s | 31 kB 00:00 2024-03-29T10:37:36.367 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:36.368 INFO:teuthology.orchestra.run.smithi173.stdout:Total 188 kB/s | 31 kB 00:00 2024-03-29T10:37:36.368 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:36.371 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:36.372 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:36.402 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:36.403 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:36.502 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:37:36.528 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:36.624 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:37.299 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:37.299 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:37.300 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:37.300 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:37.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-03-29T10:37:37.527 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:37:37.556 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:37:38.017 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:38.113 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:38.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-03-29T10:37:38.778 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:38.876 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:38.928 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:38.929 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:38.929 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:38.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-03-29T10:37:39.538 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:39.636 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:39.688 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:39.689 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:39.689 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:39.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-03-29T10:37:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:37:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:37:40.233 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:37:40.234 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:37:40.304 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:40.403 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:40.456 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:40.457 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:40.457 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:40.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-03-29T10:37:40.678 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:37:40.678 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:40.679 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:40.679 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:40.679 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:37:40.679 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:37:40.679 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:37:40.679 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:37:40.680 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:37:40.680 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:37:40.680 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:37:40.680 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:37:40.680 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:37:40.680 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:37:40.680 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:37:40.681 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:40.681 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:41.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:37:41.073 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:41.173 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:41.225 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:41.226 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:41.226 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:41.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-03-29T10:37:41.600 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:41.750 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:41.751 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T10:37:41.751 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:41.751 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T10:37:41.751 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:41.751 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:37:41.752 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:37:41.752 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:37:41.752 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:37:41.752 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:37:41.752 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:37:41.752 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:37:41.753 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:37:41.753 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:37:41.753 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:37:41.753 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:37:41.753 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:37:41.753 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:37:41.754 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:37:41.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:41.754 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:41.754 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T10:37:41.754 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-03-29T10:37:41.754 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:41.755 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.0 M 2024-03-29T10:37:41.756 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-03-29T10:37:41.756 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:41.847 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:41.936 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 24 kB 00:00 2024-03-29T10:37:41.946 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:41.961 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2024-03-29T10:37:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:41.999 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-29T10:37:41.999 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:42.019 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-29T10:37:42.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-03-29T10:37:42.053 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-29T10:37:42.087 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-29T10:37:42.104 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-29T10:37:42.121 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 165 kB/s | 52 kB 00:00 2024-03-29T10:37:42.263 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-03-29T10:37:42.279 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-29T10:37:42.355 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 867 kB/s | 203 kB 00:00 2024-03-29T10:37:42.439 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-03-29T10:37:42.464 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 321 kB/s | 59 kB 00:00 2024-03-29T10:37:42.489 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-03-29T10:37:42.493 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:42.494 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-29T10:37:42.494 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:42.506 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:42.506 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:42.596 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:42.596 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:42.613 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:42.761 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T10:37:42.762 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:42.763 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 85 k 2024-03-29T10:37:42.763 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 231 k 2024-03-29T10:37:42.763 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:42.970 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 412 kB/s | 85 kB 00:00 2024-03-29T10:37:42.971 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:42.971 INFO:teuthology.orchestra.run.smithi173.stdout:Total 409 kB/s | 85 kB 00:00 2024-03-29T10:37:42.971 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:42.978 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:42.978 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:43.034 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:43.034 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:43.110 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:43.182 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:43.275 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:37:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:43.340 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:37:43.403 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:37:43.433 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:37:43.474 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:37:43.576 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:37:43.682 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:43.693 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:37:43.759 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:37:43.983 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:43.983 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:43.983 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:43.983 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:43.983 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:43.984 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:44.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-03-29T10:37:44.471 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:37:44.524 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:37:44.557 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:37:44.605 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:37:44.628 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:37:44.855 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:45.013 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:45.013 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:45.013 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.1 M 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 13 M 2024-03-29T10:37:45.014 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:45.424 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.5 MB/s | 3.1 MB 00:00 2024-03-29T10:37:45.425 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:45.426 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.5 MB/s | 3.1 MB 00:00 2024-03-29T10:37:45.426 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:45.435 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:45.435 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:45.472 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:37:45.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:37:45.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:37:45.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:37:45.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:37:45.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:37:45.472 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:37:45.473 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:37:45.473 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:37:45.473 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:37:45.473 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:37:45.473 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:37:45.473 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:37:45.508 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:45.508 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:37:45.827 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:45.828 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:46.008 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:46.034 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:46.034 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:46.034 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:37:46.034 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:37:46.034 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:37:46.035 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:46.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:37:46.672 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:24 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:46.823 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:46.823 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T10:37:46.823 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================== 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-03-29T10:37:46.824 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:46.825 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 6.8 M 2024-03-29T10:37:46.825 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-03-29T10:37:46.825 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:47.181 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:47.182 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:47.182 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:47.182 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:47.182 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:47.182 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:47.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-03-29T10:37:47.559 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:37:47.585 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 486 kB/s | 138 kB 00:00 2024-03-29T10:37:47.974 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:35:59 AM UTC. 2024-03-29T10:37:47.976 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 367 kB/s | 248 kB 00:00 2024-03-29T10:37:48.035 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 128 kB/s | 58 kB 00:00 2024-03-29T10:37:48.124 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-29T10:37:48.124 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:48.124 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-29T10:37:48.124 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:48.124 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-29T10:37:48.124 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 171 k 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 494 k 2024-03-29T10:37:48.125 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-29T10:37:48.260 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 782 kB/s | 548 kB 00:00 2024-03-29T10:37:48.362 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 727 kB/s | 171 kB 00:00 2024-03-29T10:37:48.362 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:48.362 INFO:teuthology.orchestra.run.smithi173.stdout:Total 723 kB/s | 171 kB 00:00 2024-03-29T10:37:48.363 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-29T10:37:48.370 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-29T10:37:48.370 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-29T10:37:48.402 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 120 kB/s | 51 kB 00:00 2024-03-29T10:37:48.406 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:48.406 INFO:teuthology.orchestra.run.smithi094.stdout:Total 4.3 MB/s | 6.8 MB 00:01 2024-03-29T10:37:48.420 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:48.425 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-29T10:37:48.426 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-29T10:37:48.430 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:48.431 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:48.525 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:48.525 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:48.571 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-29T10:37:48.668 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:48.847 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:49.006 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:37:49.053 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:37:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:49.387 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:49.387 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:49.387 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-29T10:37:49.387 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:49.387 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:37:49.387 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-29T10:37:49.570 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:37:49.618 DEBUG:teuthology.parallel:result is None 2024-03-29T10:37:49.656 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:37:49.726 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:37:49.750 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:37:50.628 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:37:50.628 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:37:50.628 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:37:50.628 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:37:50.628 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:37:50.628 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:50.912 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:51.149 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-03-29T10:37:51.709 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:51.858 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:51.858 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-03-29T10:37:51.859 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:51.860 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 964 k 2024-03-29T10:37:51.860 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.8 M 2024-03-29T10:37:51.860 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:51.996 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 578 kB/s | 79 kB 00:00 2024-03-29T10:37:52.413 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 885 kB 00:00 2024-03-29T10:37:52.414 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:52.415 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.7 MB/s | 964 kB 00:00 2024-03-29T10:37:52.415 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:52.435 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:52.435 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:52.509 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:52.509 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:52.702 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:52.831 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:37:52.897 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:37:53.581 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:37:53.581 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:37:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:37:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:53.866 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:53.866 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:37:53.867 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:53.867 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:54.176 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-03-29T10:37:54.758 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:54.910 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:54.911 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:54.912 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:54.912 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-03-29T10:37:54.912 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:54.912 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 332 k 2024-03-29T10:37:54.912 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.5 M 2024-03-29T10:37:54.912 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:55.036 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 558 kB/s | 69 kB 00:00 2024-03-29T10:37:55.277 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 722 kB/s | 264 kB 00:00 2024-03-29T10:37:55.278 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:55.279 INFO:teuthology.orchestra.run.smithi094.stdout:Total 904 kB/s | 332 kB 00:00 2024-03-29T10:37:55.279 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:55.287 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:55.287 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:55.323 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:55.324 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:55.472 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:55.651 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:37:55.688 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:37:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:37:55.704 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:55.704 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:37:55.704 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:56.431 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:37:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:37:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:56.759 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:37:56.759 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:56.759 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:57.067 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-03-29T10:37:57.646 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:37:57.795 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:37:57.796 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:57.796 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T10:37:57.796 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:57.796 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 127 k 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 456 k 2024-03-29T10:37:57.797 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:37:58.109 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 406 kB/s | 127 kB 00:00 2024-03-29T10:37:58.110 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:58.110 INFO:teuthology.orchestra.run.smithi094.stdout:Total 404 kB/s | 127 kB 00:00 2024-03-29T10:37:58.110 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:37:58.115 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:37:58.115 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:37:58.171 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:37:58.172 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:37:58.311 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:37:58.408 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:58.865 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:59.153 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:37:59.153 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:59.154 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:37:59.154 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:59.154 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:37:59.154 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:37:59.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-03-29T10:37:59.944 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:00.045 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:00.101 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:00.102 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:38:00.102 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:00.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-03-29T10:38:00.727 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:00.877 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T10:38:00.878 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:00.879 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 31 k 2024-03-29T10:38:00.879 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 140 k 2024-03-29T10:38:00.879 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:38:01.146 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 117 kB/s | 31 kB 00:00 2024-03-29T10:38:01.147 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:01.147 INFO:teuthology.orchestra.run.smithi094.stdout:Total 116 kB/s | 31 kB 00:00 2024-03-29T10:38:01.147 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:38:01.151 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:38:01.151 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:38:01.178 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:38:01.178 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:38:01.288 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:38:01.394 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:01.795 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:02.067 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:02.067 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:02.067 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:38:02.067 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:02.067 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:02.067 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:02.244 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-03-29T10:38:02.804 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:02.901 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:02.955 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:02.956 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:38:02.956 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:03.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-03-29T10:38:03.564 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:03.660 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:03.711 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:03.712 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:38:03.712 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:03.770 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-03-29T10:38:04.333 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:04.430 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:04.482 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:04.482 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:38:04.483 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:04.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-03-29T10:38:05.092 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:05.190 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:05.241 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:05.242 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:38:05.242 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:05.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-03-29T10:38:05.869 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:05.966 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:06.017 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:06.018 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:38:06.018 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:06.070 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-03-29T10:38:06.636 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:06.734 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:06.785 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:06.786 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-03-29T10:38:06.787 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:06.840 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-03-29T10:38:07.423 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:07.582 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:07.583 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:07.583 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-29T10:38:07.583 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:07.583 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:38:07.583 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:38:07.583 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:07.584 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:38:07.584 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:07.584 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T10:38:07.584 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:07.584 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 85 k 2024-03-29T10:38:07.584 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 231 k 2024-03-29T10:38:07.584 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:38:08.042 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 186 kB/s | 85 kB 00:00 2024-03-29T10:38:08.043 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:08.043 INFO:teuthology.orchestra.run.smithi094.stdout:Total 185 kB/s | 85 kB 00:00 2024-03-29T10:38:08.043 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:38:08.050 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:38:08.050 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:38:08.105 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:38:08.106 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:38:08.246 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:38:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:08.775 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:09.072 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:09.072 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:09.072 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:38:09.072 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:09.072 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:09.072 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:09.287 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-03-29T10:38:09.860 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:10.016 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:10.017 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T10:38:10.018 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:10.018 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.1 M 2024-03-29T10:38:10.018 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 13 M 2024-03-29T10:38:10.018 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:38:11.871 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.7 MB/s | 3.1 MB 00:01 2024-03-29T10:38:11.871 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:11.872 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.7 MB/s | 3.1 MB 00:01 2024-03-29T10:38:11.872 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:38:11.883 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:38:11.883 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:38:11.959 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:38:11.959 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:38:12.399 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:38:12.463 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:12.488 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:12.488 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:12.488 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:38:12.488 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:38:12.488 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:38:12.488 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:13.583 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:13.927 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:13.928 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:38:13.928 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:13.928 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:13.928 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:13.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-03-29T10:38:14.502 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 10:36:22 AM UTC. 2024-03-29T10:38:14.652 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-03-29T10:38:14.653 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:14.653 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-03-29T10:38:14.653 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:14.653 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-03-29T10:38:14.653 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:38:14.653 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:14.654 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-03-29T10:38:14.654 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-03-29T10:38:14.654 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-03-29T10:38:14.654 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:14.654 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 171 k 2024-03-29T10:38:14.654 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 494 k 2024-03-29T10:38:14.654 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-03-29T10:38:14.975 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 534 kB/s | 171 kB 00:00 2024-03-29T10:38:14.976 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:14.976 INFO:teuthology.orchestra.run.smithi094.stdout:Total 531 kB/s | 171 kB 00:00 2024-03-29T10:38:14.976 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-03-29T10:38:14.984 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-03-29T10:38:14.984 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-03-29T10:38:15.040 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-03-29T10:38:15.040 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-03-29T10:38:15.181 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-03-29T10:38:15.291 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:15.741 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:16.021 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:16.021 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:16.021 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-03-29T10:38:16.022 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:16.022 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:16.022 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-03-29T10:38:16.245 DEBUG:teuthology.parallel:result is None 2024-03-29T10:38:16.246 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:38:16.469 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:38:16.489 INFO:teuthology.orchestra.run.smithi094.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:16.490 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:16.490 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:38:16.492 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:38:16.629 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:38:16.654 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:16.654 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:16.654 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:38:16.656 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:38:16.656 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:16.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:38:16.683 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:16.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:38:16.725 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:38:16.726 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:16.726 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:38:16.752 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:38:16.818 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:16.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:38:16.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:38:16.918 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:38:16.918 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:16.918 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:38:16.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:38:17.012 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:17.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:38:17.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:38:17.109 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:38:17.109 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:17.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:38:17.136 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:38:17.203 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:17.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:38:17.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:38:17.298 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:38:17.382 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:38:17.383 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2024-03-29T10:38:17.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2024-03-29T10:38:17.412 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:38:17.413 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:38:17.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:38:17.479 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:38:17.480 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:38:17.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:38:17.557 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:38:17.557 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:38:17.557 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622646', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBET4uWJ1dRCY8i6GpiJNqX+aKDBuXu7e+8lWovKubi62FhnlCX+Wyv2xAdWjc9TNzP2I0g6FSbsqwBbCUYvKKv4=', 'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA0TujK83NAzIr2yYFWZMtxTPs5Qz3YLnVMBZnma4ddeq0wqf85ZCyNNfWNT6kIfQwZeb7neKme59t5iHy5j8Oo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:38:17.557 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:38:17.585 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:38:17.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:38:17.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:38:17.669 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:17.669 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:38:17.720 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:38:17.721 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 10:38:12.765361496 +0000 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 10:35:08.046505216 +0000 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 10:35:08.046505216 +0000 2024-03-29T10:38:17.776 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 10:35:08.046505216 +0000 2024-03-29T10:38:17.776 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:38:17.841 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T10:38:17.841 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T10:38:17.841 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000101945 s, 5.0 MB/s 2024-03-29T10:38:17.842 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:38:17.898 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-03-29T10:38:17.954 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:38:17.954 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:17.954 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T10:38:17.954 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:17.954 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:17.954 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 10:38:12.765361496 +0000 2024-03-29T10:38:17.955 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 10:35:08.382499451 +0000 2024-03-29T10:38:17.955 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 10:35:08.383499434 +0000 2024-03-29T10:38:17.955 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 10:35:08.382499451 +0000 2024-03-29T10:38:17.955 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:38:18.018 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T10:38:18.019 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T10:38:18.019 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.4289e-05 s, 6.1 MB/s 2024-03-29T10:38:18.020 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:38:18.075 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 10:38:12.766361479 +0000 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 10:35:08.709493841 +0000 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 10:35:08.709493841 +0000 2024-03-29T10:38:18.131 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 10:35:08.709493841 +0000 2024-03-29T10:38:18.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:38:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T10:38:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T10:38:18.196 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 8.0475e-05 s, 6.4 MB/s 2024-03-29T10:38:18.197 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:38:18.252 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-03-29T10:38:18.306 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:38:18.306 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:18.306 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:38:18.306 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:18.307 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:18.307 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-03-29 10:38:12.766361479 +0000 2024-03-29T10:38:18.307 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-03-29 10:35:09.026488402 +0000 2024-03-29T10:38:18.307 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-03-29 10:35:09.026488402 +0000 2024-03-29T10:38:18.307 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-03-29 10:35:09.026488402 +0000 2024-03-29T10:38:18.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:38:18.372 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-03-29T10:38:18.372 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-03-29T10:38:18.373 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000104243 s, 4.9 MB/s 2024-03-29T10:38:18.374 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:38:18.430 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:18.430 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:18.430 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:38:18.445 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:38:18.446 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-03-29T10:38:18.501 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:38:18.501 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:18.501 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:38:18.501 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:18.501 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:18.501 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 10:37:46.317721992 +0000 2024-03-29T10:38:18.502 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 10:35:14.972565419 +0000 2024-03-29T10:38:18.502 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 10:35:14.972565419 +0000 2024-03-29T10:38:18.502 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 10:35:14.972565419 +0000 2024-03-29T10:38:18.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:38:18.566 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T10:38:18.567 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T10:38:18.567 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000257312 s, 2.0 MB/s 2024-03-29T10:38:18.568 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:38:18.624 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-03-29T10:38:18.679 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:38:18.680 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:38:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 10:37:46.318721972 +0000 2024-03-29T10:38:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 10:35:15.308558502 +0000 2024-03-29T10:38:18.680 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 10:35:15.308558502 +0000 2024-03-29T10:38:18.681 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 10:35:15.308558502 +0000 2024-03-29T10:38:18.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:38:18.745 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T10:38:18.746 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T10:38:18.746 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00021061 s, 2.4 MB/s 2024-03-29T10:38:18.747 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:38:18.803 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 10:37:46.318721972 +0000 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 10:35:15.648551502 +0000 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 10:35:15.648551502 +0000 2024-03-29T10:38:18.858 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 10:35:15.648551502 +0000 2024-03-29T10:38:18.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:38:18.923 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T10:38:18.923 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T10:38:18.924 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000208861 s, 2.5 MB/s 2024-03-29T10:38:18.925 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:38:18.980 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-03-29T10:38:19.034 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-29 10:37:46.318721972 +0000 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-29 10:35:16.007544112 +0000 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-29 10:35:16.007544112 +0000 2024-03-29T10:38:19.035 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-29 10:35:16.007544112 +0000 2024-03-29T10:38:19.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:38:19.101 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-29T10:38:19.101 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-29T10:38:19.101 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000199393 s, 2.6 MB/s 2024-03-29T10:38:19.102 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:38:19.157 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:19.157 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:38:19.157 INFO:tasks.ceph:Generating config... 2024-03-29T10:38:19.158 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:38:19.159 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:38:19.159 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:38:19.159 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:38:19.159 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:38:19.159 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:38:19.159 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:38:19.159 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:38:19.159 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:38:19.159 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:38:19.159 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:38:19.159 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:38:19.159 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:38:19.159 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:38:19.159 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:38:19.159 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:38:19.159 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:38:19.160 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:38:19.160 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:38:19.160 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:38:19.160 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:38:19.160 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:38:19.160 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:38:19.161 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:38:19.162 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:38:19.162 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:38:19.162 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:38:19.162 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:38:19.212 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:38:19.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:38:19.258 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:19.325 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.94:6789]'), ('mon.c', '[v1:172.21.15.94:6790]'), ('mon.b', '[v1:172.21.15.173:6789]')] 2024-03-29T10:38:19.325 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.94:6789],[v1:172.21.15.94:6790],[v1:172.21.15.173:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:38:19.326 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:19.326 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:38:19.379 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.94:6789]' --addv c '[v1:172.21.15.94:6790]' --addv b '[v1:172.21.15.173:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:19.464 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:19.464 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: generated fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:19.464 INFO:teuthology.orchestra.run.smithi094.stdout:setting min_mon_release = quincy 2024-03-29T10:38:19.464 INFO:teuthology.orchestra.run.smithi094.stdout:epoch 0 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:min_mon_release 17 (quincy) 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:election_strategy: 1 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:19.465 INFO:teuthology.orchestra.run.smithi094.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:38:19.466 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:38:19.481 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 86a40404-ccd6-4767-bb34-c1deee5abdf7... 2024-03-29T10:38:19.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:38:19.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:38:19.570 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2024-03-29T10:38:19.571 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:38:19.572 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.573 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "[v1:172.21.15.94:6789],[v1:172.21.15.94:6790],[v1:172.21.15.173:6789]" 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: lockdep = True 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: mon client directed command retry = 5 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: mon election default strategy = 1 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: ms bind msgr2 = False 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: ms inject socket failures = 5000 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: ms type = async 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2024-03-29T10:38:19.574 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug trim objects = true 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2024-03-29T10:38:19.575 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression algorithm = zstd 2024-03-29T10:38:19.576 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore compression mode = aggressive 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore zero block detection = True 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 20 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 20 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 20 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 10 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: osd class default list = * 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: osd class load list = * 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: osd client watch timeout = 120 2024-03-29T10:38:19.577 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify cached snaps = True 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify missing on start = True 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock override recovery settings = True 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue = debug_random 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: osd op queue cut off = debug_random 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: osd shutdown pgref assert = True 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-03-29T10:38:19.578 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:19.579 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.580 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: mon min osdmap epochs = 50 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune interval = 2 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune min = 15 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: mon scrub interval = 300 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on pool no app = False 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: paxos service trim min = 10 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:38:19.581 INFO:teuthology.orchestra.run.smithi094.stdout: debug monc = 20 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi094.stdout: debug objecter = 20 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi094.stdout: debug rados = 20 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2024-03-29T10:38:19.582 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.583 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2024-03-29T10:38:19.584 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = "[v1:172.21.15.94:6789],[v1:172.21.15.94:6790],[v1:172.21.15.173:6789]" 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: lockdep = True 2024-03-29T10:38:19.585 INFO:teuthology.orchestra.run.smithi173.stdout: mon client directed command retry = 5 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: mon election default strategy = 1 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: ms bind msgr2 = False 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject socket failures = 5000 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: ms type = async 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:38:19.586 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug trim objects = true 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.587 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression algorithm = zstd 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore compression mode = aggressive 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore zero block detection = True 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 20 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 20 2024-03-29T10:38:19.588 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 20 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 10 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd class default list = * 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd class load list = * 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd client watch timeout = 120 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify cached snaps = True 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify missing on start = True 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock override recovery settings = True 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2024-03-29T10:38:19.589 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue = debug_random 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue cut off = debug_random 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: osd shutdown pgref assert = True 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:19.590 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:38:19.591 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: mon min osdmap epochs = 50 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune interval = 2 2024-03-29T10:38:19.592 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune min = 15 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub interval = 300 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on pool no app = False 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: paxos service trim min = 10 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 20 2024-03-29T10:38:19.593 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-29T10:38:19.594 INFO:teuthology.orchestra.run.smithi173.stdout: debug objecter = 20 2024-03-29T10:38:19.594 INFO:teuthology.orchestra.run.smithi173.stdout: debug rados = 20 2024-03-29T10:38:19.594 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2024-03-29T10:38:19.594 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.c] 2024-03-29T10:38:19.594 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.b] 2024-03-29T10:38:19.595 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:38:19.595 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:38:19.650 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:38:19.651 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:19.651 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:38:19.665 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:19.666 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:38:19.721 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:38:19.722 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:19.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:38:19.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:19.801 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:19.801 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:19.855 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:38:19.856 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:19.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:38:19.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:19.895 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:19.895 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:19.949 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:38:19.950 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:38:19.950 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:38:20.004 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:38:20.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:38:20.069 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:38:20.073 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:38:20.073 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:38:20.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:38:20.116 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:38:20.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:38:20.173 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:38:20.188 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:38:20.189 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:38:20.189 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:20.215 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:20.215 INFO:tasks.ceph:role: osd.0 2024-03-29T10:38:20.215 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:38:20.216 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:38:20.282 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:20.282 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:20.282 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:20.282 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:20.282 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:20.282 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:20.282 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:20.283 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:20.283 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:20.283 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:20.321 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T10:38:20.323 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T10:38:20.323 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:20.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:20.466 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:20.532 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:20.532 INFO:tasks.ceph:role: osd.1 2024-03-29T10:38:20.533 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:38:20.533 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:38:20.599 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:20.599 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:20.599 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:20.600 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:20.600 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:20.600 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:20.600 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:20.639 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T10:38:20.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T10:38:20.641 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:20.712 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:20.784 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:20.851 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:20.851 INFO:tasks.ceph:role: osd.2 2024-03-29T10:38:20.851 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:38:20.851 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:38:20.917 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:20.918 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:20.958 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T10:38:20.959 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T10:38:20.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:21.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:21.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:21.176 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:21.176 INFO:tasks.ceph:role: osd.3 2024-03-29T10:38:21.176 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:38:21.176 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:38:21.242 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:21.242 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:21.242 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:21.243 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:21.243 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:21.243 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:21.243 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:21.282 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2024-03-29T10:38:21.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2024-03-29T10:38:21.284 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:21.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:21.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:21.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:21.535+0000 7f8815415640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:38:21.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:21.535+0000 7f8815415640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:38:21.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:21.535+0000 7f8815415640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:38:21.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:21.535+0000 7f8815415640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:38:21.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:21.535+0000 7f8815415640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:38:21.520 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:21.536+0000 7f8815415640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:38:24.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:24.386 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:24.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:24.492+0000 7f5e4fd0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:38:24.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:24.492+0000 7f5e4fd0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:38:24.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:24.492+0000 7f5e4fd0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:38:24.476 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:24.492+0000 7f5e4fd0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:38:24.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:24.492+0000 7f5e4fd0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:38:24.477 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:24.492+0000 7f5e4fd0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:38:27.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:27.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:27.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:27.462+0000 7f064d923640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:38:27.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:27.462+0000 7f064d923640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:38:27.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:27.463+0000 7f064d923640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:38:27.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:27.463+0000 7f064d923640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:38:27.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:27.463+0000 7f064d923640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:38:27.447 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:27.463+0000 7f064d923640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:38:30.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:30.319 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:30.411 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:30.427+0000 7fd8407bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:38:30.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:30.427+0000 7fd8407bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:38:30.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:30.427+0000 7fd8407bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:38:30.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:30.427+0000 7fd8407bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:38:30.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:30.427+0000 7fd8407bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:38:30.412 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:30.427+0000 7fd8407bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:38:33.259 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:33.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:33.321 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:33.321 INFO:tasks.ceph:role: osd.4 2024-03-29T10:38:33.321 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:38:33.321 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:38:33.398 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:33.399 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:33.906 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T10:38:33.908 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T10:38:33.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:34.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:34.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:38:34.170 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:34.170 INFO:tasks.ceph:role: osd.5 2024-03-29T10:38:34.170 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:38:34.171 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:38:34.246 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:34.247 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:34.699 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T10:38:34.701 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T10:38:34.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:38:34.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:38:34.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:38:34.901 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:34.901 INFO:tasks.ceph:role: osd.6 2024-03-29T10:38:34.902 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:38:34.902 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:38:34.978 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:34.978 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:34.978 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:34.979 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:34.979 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:35.477 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T10:38:35.478 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T10:38:35.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:38:35.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:38:35.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:38:35.717 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:35.717 INFO:tasks.ceph:role: osd.7 2024-03-29T10:38:35.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:38:35.718 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:38:35.792 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:35.792 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:35.792 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:35.792 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:35.792 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:35.792 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:35.792 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:35.793 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:35.793 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:35.793 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:36.294 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-29T10:38:36.296 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-29T10:38:36.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:38:36.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:38:36.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:36.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:36.588+0000 7f62cf213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:38:36.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:36.588+0000 7f62cf213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:38:36.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:36.588+0000 7f62cf213640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:38:36.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:36.588+0000 7f62cf213640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:38:36.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:36.588+0000 7f62cf213640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:38:36.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:36.589+0000 7f62cf213640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:38:39.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:38:39.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:39.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:39.560+0000 7f9c99419640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:38:39.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:39.560+0000 7f9c99419640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:38:39.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:39.560+0000 7f9c99419640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:38:39.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:39.560+0000 7f9c99419640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:38:39.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:39.560+0000 7f9c99419640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:38:39.545 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:39.561+0000 7f9c99419640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:38:42.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:38:42.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:42.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:42.538+0000 7f948a573640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:38:42.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:42.538+0000 7f948a573640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:38:42.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:42.539+0000 7f948a573640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:38:42.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:42.539+0000 7f948a573640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:38:42.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:42.539+0000 7f948a573640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:38:42.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:42.539+0000 7f948a573640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:38:45.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:38:45.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:45.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:45.517+0000 7f792d2b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:38:45.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:45.518+0000 7f792d2b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:38:45.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:45.518+0000 7f792d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:38:45.502 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:45.518+0000 7f792d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:38:45.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:45.518+0000 7f792d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:38:45.503 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:45.518+0000 7f792d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:38:48.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:38:48.399 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:38:48.400 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:48.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:38:48.432 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:48.433 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:38:48.502 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:48.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:38:48.572 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:48.572 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:38:48.640 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:48.640 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:38:48.708 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:48.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:38:48.740 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:48.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:38:48.810 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:48.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:38:48.879 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:48.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:38:48.948 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:48.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:38:49.016 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:49.016 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:38:49.032 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:49.032 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:38:49.071 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:38:49.072 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:38:49.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:38:49.115 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.y] 2024-03-29T10:38:49.115 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAcmgZmBbVEARAA5rCM5ONVT+8VakSDIBA2TA== 2024-03-29T10:38:49.115 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2024-03-29T10:38:49.115 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAdmgZmPnb6HxAARDZ19p6ttf6MnVujBS7SOQ== 2024-03-29T10:38:49.115 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAgmgZmPC1hHRAA6/g/UNM4EO3lxvR/LstaSQ== 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAjmgZmbjWjGxAAlfX5OOHiNETRfhpJ6PTKlw== 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAmmgZmnaeEGRAAqBw74R/YiWSYTcwiF+Nf9A== 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAcmgZmbN8oBRAALy34fTEFwP0Vwc7jRN+YwQ== 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.4] 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAsmgZmze8jIxAAhu/E1Wjk16EwdtWcE82HlA== 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.5] 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAvmgZmnd14IRAANp4df9jn/XdtjQC1Yq5Y1w== 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.6] 2024-03-29T10:38:49.116 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAymgZm4BIrIBAAg9+cBGTESir2tW2ZMIaqtQ== 2024-03-29T10:38:49.117 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.7] 2024-03-29T10:38:49.117 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQA1mgZmCPPuHhAAdl/EQqXy3VRJsN3JTYu/JA== 2024-03-29T10:38:49.117 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2024-03-29T10:38:49.117 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAcmgZmhrXuBxAAc2kh+oHbNkrUeh6rlP+kJQ== 2024-03-29T10:38:49.117 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2024-03-29T10:38:49.117 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQAcmgZmRqlZCxAAf22XJGG1pCh9pwudK+sYWA== 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.y] 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAcmgZmBbVEARAA5rCM5ONVT+8VakSDIBA2TA== 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.0] 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAdmgZmPnb6HxAARDZ19p6ttf6MnVujBS7SOQ== 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.1] 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAgmgZmPC1hHRAA6/g/UNM4EO3lxvR/LstaSQ== 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.2] 2024-03-29T10:38:49.138 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAjmgZmbjWjGxAAlfX5OOHiNETRfhpJ6PTKlw== 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.3] 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAmmgZmnaeEGRAAqBw74R/YiWSYTcwiF+Nf9A== 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.x] 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAcmgZmbN8oBRAALy34fTEFwP0Vwc7jRN+YwQ== 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.4] 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAsmgZmze8jIxAAhu/E1Wjk16EwdtWcE82HlA== 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.5] 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAvmgZmnd14IRAANp4df9jn/XdtjQC1Yq5Y1w== 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.6] 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAymgZm4BIrIBAAg9+cBGTESir2tW2ZMIaqtQ== 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.7] 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA1mgZmCPPuHhAAdl/EQqXy3VRJsN3JTYu/JA== 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout:[client.0] 2024-03-29T10:38:49.139 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAcmgZmhrXuBxAAc2kh+oHbNkrUeh6rlP+kJQ== 2024-03-29T10:38:49.140 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-03-29T10:38:49.140 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQAcmgZmRqlZCxAAf22XJGG1pCh9pwudK+sYWA== 2024-03-29T10:38:49.140 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:38:49.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:38:49.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.396 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.450 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:38:49.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:38:49.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.557 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.665 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:38:49.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:38:49.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:38:49.775 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:38:49.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:38:49.832 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:38:49.832 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:38:49.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:38:49.955 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:38:49.981 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:38:50.049 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:38:50.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:38:50.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:38:50.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:38:50.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:38:50.324 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:50.326 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:50.378 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:38:50.378 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:38:50.379 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:38:50.381 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:38:50.381 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:38:50.381 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:38:50.383 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:38:50.383 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:38:50.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:38:50.421 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:38:50.421 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:38:50.421 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:38:50.422 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:38:50.424 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:38:50.424 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:38:50.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:38:50.426 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:38:50.426 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:38:50.426 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:38:50.525 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.541+0000 7fc730dcf640 10 monclient: get_monmap_and_config 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.541+0000 7fc730dcf640 10 monclient: build_initial_monmap 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.541+0000 7fc730dcf640 10 monclient: monmap: 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:50.542807+0000 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:50.542807+0000 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:50.526 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:50.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.542+0000 7fc730dcf640 1 Processor -- start 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.542+0000 7fc730dcf640 1 -- start start 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.542+0000 7fc730dcf640 10 monclient: init 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc72c0abf30 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc72c140340 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc72c148930 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c0aca20 con 0x7fc72c0abf30 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c0ad910 con 0x7fc72c148930 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c13fe20 con 0x7fc72c140340 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc730dcf640 10 monclient(hunting): authenticate will time out at 781.811334s 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc72a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc72c0abf30 0x7fc72c0ac310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59214/0 (socket says 172.21.15.94:59214) 2024-03-29T10:38:50.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:50.543+0000 7fc72a59c640 1 -- 172.21.15.94:0/1656831179 learned_addr learned my addr 172.21.15.94:0/1656831179 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient(hunting): tick 2024-03-29T10:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:38:53.527 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 1 monclient(hunting): continuing hunt 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 1 -- 172.21.15.94:0/1656831179 >> v1:172.21.15.94:6790/0 conn(0x7fc72c140340 legacy=0x7fc72c1407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 1 -- 172.21.15.94:0/1656831179 >> v1:172.21.15.173:6789/0 conn(0x7fc72c148930 legacy=0x7fc72c14ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 1 -- 172.21.15.94:0/1656831179 >> v1:172.21.15.94:6789/0 conn(0x7fc72c0abf30 legacy=0x7fc72c0ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient(hunting): picked mon.noname-c con 0x7fc70c0036e0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient(hunting): picked mon.noname-a con 0x7fc70c0072d0 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient(hunting): picked mon.noname-b con 0x7fc70c00b000 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 1 -- 172.21.15.94:0/1656831179 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c0ae150 con 0x7fc70c0072d0 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 1 -- 172.21.15.94:0/1656831179 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70c00fdf0 con 0x7fc70c0036e0 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 1 -- 172.21.15.94:0/1656831179 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc70c011030 con 0x7fc70c00b000 2024-03-29T10:38:53.528 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:53.543+0000 7fc728d99640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.708+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737182521 0 0) 0x7fc72c0ae150 con 0x7fc70c0072d0 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.708+0000 7fc729d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.708+0000 7fc729d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.708+0000 7fc729d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.708+0000 7fc729d9b640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.708+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc718003520 con 0x7fc70c0072d0 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799878080 0 0) 0x7fc70c011030 con 0x7fc70c00b000 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:55.694 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc72c0ae150 con 0x7fc70c00b000 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989806747 0 0) 0x7fc70c00fdf0 con 0x7fc70c0036e0 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70c011030 con 0x7fc70c0036e0 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199208408 0 0) 0x7fc72c0ae150 con 0x7fc70c00b000 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 >> v1:172.21.15.173:6789/0 conn(0x7fc70c0036e0 legacy=0x7fc70c005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 >> v1:172.21.15.94:6789/0 conn(0x7fc70c0072d0 legacy=0x7fc70c009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 monclient: found mon.noname-b 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:38:55.695 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c012270 con 0x7fc70c00b000 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: _finish_auth 0 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc7240019d0 con 0x7fc70c00b000 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: dump: 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:55.696 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc730dcf640 5 monclient: authenticate success, global_id 4106 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc730dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc724001b30 con 0x7fc70c00b000 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 1 -- 172.21.15.94:0/1656831179 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc724001f80 con 0x7fc70c00b000 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc729d9b640 10 monclient: dump: 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.697 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc730dcf640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:55.698 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.709+0000 7fc730dcf640 4 set_mon_vals no callback set 2024-03-29T10:38:55.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.710+0000 7fc730dcf640 10 monclient: shutdown 2024-03-29T10:38:55.703 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.710+0000 7fc730dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.710+0000 7fc730dcf640 1 -- 172.21.15.94:0/1656831179 >> v1:172.21.15.94:6790/0 conn(0x7fc70c00b000 legacy=0x7fc70c00d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.710+0000 7fc730dcf640 1 -- 172.21.15.94:0/1656831179 shutdown_connections 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.710+0000 7fc730dcf640 1 -- 172.21.15.94:0/1656831179 >> 172.21.15.94:0/1656831179 conn(0x7fc72c0b3750 msgr2=0x7fc72c0b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.710+0000 7fc730dcf640 1 -- 172.21.15.94:0/1656831179 shutdown_connections 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.710+0000 7fc730dcf640 1 -- 172.21.15.94:0/1656831179 wait complete. 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.712+0000 7fc730dcf640 10 monclient: build_initial_monmap 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.712+0000 7fc730dcf640 10 monclient: monmap: 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:55.704 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.712+0000 7fc730dcf640 1 librados: starting msgr at 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.712+0000 7fc730dcf640 1 librados: starting objecter 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.712+0000 7fc730dcf640 1 Processor -- start 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.713+0000 7fc730dcf640 1 -- start start 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.713+0000 7fc730dcf640 1 librados: setting wanted keys 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.713+0000 7fc730dcf640 1 librados: calling monclient init 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.713+0000 7fc730dcf640 10 monclient: init 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.713+0000 7fc730dcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.713+0000 7fc730dcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.713+0000 7fc730dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc70c0036e0 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc730dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc70c0072d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc730dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc70c00b000 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:55.705 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc730dcf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c1419b0 con 0x7fc70c00b000 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc730dcf640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c1d5300 con 0x7fc70c0072d0 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc730dcf640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72c1d4fd0 con 0x7fc70c0036e0 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc730dcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc730dcf640 10 monclient(hunting): authenticate will time out at 786.982174s 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc72ad9d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc70c00b000 0x7fc72c070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51056/0 (socket says 172.21.15.94:51056) 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc72ad9d640 1 -- 172.21.15.94:0/638747186 learned_addr learned my addr 172.21.15.94:0/638747186 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 client.?.objecter ms_handle_connect 0x7fc70c00b000 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 client.?.objecter ms_handle_connect 0x7fc70c0036e0 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 client.?.objecter ms_handle_connect 0x7fc70c0072d0 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473822011 0 0) 0x7fc72c1419b0 con 0x7fc70c00b000 2024-03-29T10:38:55.706 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): my global_id is 4110 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc70c00e4e0 con 0x7fc70c00b000 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2563228478 0 0) 0x7fc70c00e4e0 con 0x7fc70c00b000 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc72c1419b0 con 0x7fc70c00b000 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc724001e80 con 0x7fc70c00b000 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): dump: 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.707 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.714+0000 7fc7237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2293509764 0 0) 0x7fc72c1419b0 con 0x7fc70c00b000 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 >> v1:172.21.15.94:6790/0 conn(0x7fc70c0036e0 legacy=0x7fc72c1d5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 >> v1:172.21.15.173:6789/0 conn(0x7fc70c0072d0 legacy=0x7fc72c1359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 monclient: found mon.a 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c1360c0 con 0x7fc70c00b000 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: _finish_auth 0 2024-03-29T10:38:55.708 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 5 monclient: authenticate success, global_id 4110 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 10 monclient: _renew_subs 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c136310 con 0x7fc70c00b000 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 10 monclient: _renew_subs 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c071210 con 0x7fc70c00b000 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc730dcf640 1 librados: init done 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc724003c40 con 0x7fc70c00b000 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc72959a640 4 set_mon_vals no callback set 2024-03-29T10:38:55.709 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc724004090 con 0x7fc70c00b000 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 monclient: dump: 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:55.710 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc724004600 con 0x7fc70c00b000 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4070523823 0 0) 0x7fc7240069d0 con 0x7fc70c00b000 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 10 client.4110.objecter ms_dispatch 0x7fc72c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.715+0000 7fc7237fe640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.716+0000 7fc730dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.716+0000 7fc730dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.716+0000 7fc730dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.716+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc710005080 con 0x7fc70c00b000 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.718+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc724005b20 con 0x7fc70c00b000 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.718+0000 7fc7237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:55.711 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.718+0000 7fc7237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.790+0000 7fc730dcf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:38:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.791+0000 7fc730dcf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:38:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.791+0000 7fc730dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:55.775 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.791+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc710005330 con 0x7fc70c00b000 2024-03-29T10:38:55.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.792+0000 7fc7237fe640 1 -- 172.21.15.94:0/638747186 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fc724005e20 con 0x7fc70c00b000 2024-03-29T10:38:55.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.792+0000 7fc7237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:38:55.776 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.792+0000 7fc7237fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:38:55.776 INFO:teuthology.orchestra.run.smithi094.stderr:adjusted tunables profile to default 2024-03-29T10:38:55.777 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 10 librados: watch_flush enter 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 10 librados: watch_flush exit 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 10 client.4110.objecter successfully canceled tick 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 10 monclient: shutdown 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 >> v1:172.21.15.94:6789/0 conn(0x7fc70c00b000 legacy=0x7fc72c070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 shutdown_connections 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 >> 172.21.15.94:0/638747186 conn(0x7fc72c0b3750 msgr2=0x7fc72c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 shutdown_connections 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 1 -- 172.21.15.94:0/638747186 wait complete. 2024-03-29T10:38:55.778 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:55.793+0000 7fc730dcf640 1 librados: shutdown 2024-03-29T10:38:55.789 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:38:55.789 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:38:55.789 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:55.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:38:55.817 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:55.817 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:38:55.883 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:55.883 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:38:55.950 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-03-29T10:38:55.950 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:38:56.017 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:56.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:38:56.044 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:56.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:38:56.112 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:56.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:38:56.174 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:56.191+0000 7fdf2ea20040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:38:56.179 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-29T10:38:56.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:38:56.191 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:56.207+0000 7f6834a13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:38:56.230 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:56.246+0000 7fdf2ea20040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:38:56.244 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:56.260+0000 7f6834a13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:38:56.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new a30ab5d7-7189-4c97-81b1-72012cadfce5 0 2024-03-29T10:38:56.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.403+0000 7f8fb0206640 10 monclient: get_monmap_and_config 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.403+0000 7f8fb0206640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.403+0000 7f8fb0206640 10 monclient: monmap: 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:56.404688+0000 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:56.404688+0000 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:56.388 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.389 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.404+0000 7f8fb0206640 1 Processor -- start 2024-03-29T10:38:56.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.404+0000 7f8fb0206640 1 -- start start 2024-03-29T10:38:56.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.404+0000 7f8fb0206640 10 monclient: init 2024-03-29T10:38:56.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa809ac40 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa809c870 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa80b4360 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa809dec0 con 0x7f8fa80b4360 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa809da30 con 0x7f8fa809ac40 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80b39c0 con 0x7f8fa809c870 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient(hunting): authenticate will time out at 783.655005s 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fadfa2640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f8fa809ac40 0x7f8fa80b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34228/0 (socket says 172.21.15.173:34228) 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fadfa2640 1 -- 172.21.15.173:0/3053418753 learned_addr learned my addr 172.21.15.173:0/3053418753 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522696759 0 0) 0x7f8fa809da30 con 0x7f8fa809ac40 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94003520 con 0x7f8fa809ac40 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503782739 0 0) 0x7f8fa0001490 con 0x7f8fa80b4360 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa809da30 con 0x7f8fa80b4360 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3394538276 0 0) 0x7f8f94003520 con 0x7f8fa809ac40 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 >> v1:172.21.15.94:6790/0 conn(0x7f8fa809c870 legacy=0x7f8fa813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 >> v1:172.21.15.94:6789/0 conn(0x7f8fa80b4360 legacy=0x7f8fa814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.399 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 monclient: found mon.noname-c 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80b5160 con 0x7f8fa809ac40 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8fa4001690 con 0x7f8fa809ac40 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: dump: 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fa40017f0 con 0x7f8fa809ac40 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8fb0206640 4 set_mon_vals no callback set 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 1 -- 172.21.15.173:0/3053418753 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8fa4001c00 con 0x7f8fa809ac40 2024-03-29T10:38:56.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.405+0000 7f8facfa0640 10 monclient: dump: 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.406+0000 7f8fb0206640 10 monclient: shutdown 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.406+0000 7f8fb0206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.406+0000 7f8fb0206640 1 -- 172.21.15.173:0/3053418753 >> v1:172.21.15.173:6789/0 conn(0x7f8fa809ac40 legacy=0x7f8fa80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.406+0000 7f8fb0206640 1 -- 172.21.15.173:0/3053418753 shutdown_connections 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.406+0000 7f8fb0206640 1 -- 172.21.15.173:0/3053418753 >> 172.21.15.173:0/3053418753 conn(0x7f8fa8066b00 msgr2=0x7f8fa8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.406+0000 7f8fb0206640 1 -- 172.21.15.173:0/3053418753 shutdown_connections 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.406+0000 7f8fb0206640 1 -- 172.21.15.173:0/3053418753 wait complete. 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient: monmap: 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.403 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 librados: starting msgr at 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 librados: starting objecter 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 Processor -- start 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 -- start start 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 librados: setting wanted keys 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 librados: calling monclient init 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient: init 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa809ac40 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa80b0540 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa81d7b50 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80b0ae0 con 0x7f8fa81d7b50 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80afeb0 con 0x7f8fa80b0540 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa80b1640 con 0x7f8fa809ac40 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.407+0000 7f8fb0206640 10 monclient(hunting): authenticate will time out at 783.657841s 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fad7a1640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f8fa80b0540 0x7f8fa81d7440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34242/0 (socket says 172.21.15.173:34242) 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fad7a1640 1 -- 172.21.15.173:0/2802710639 learned_addr learned my addr 172.21.15.173:0/2802710639 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 client.?.objecter ms_handle_connect 0x7f8fa80b0540 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 client.?.objecter ms_handle_connect 0x7f8fa809ac40 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 client.?.objecter ms_handle_connect 0x7f8fa81d7b50 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.405 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624205543 0 0) 0x7f8fa80afeb0 con 0x7f8fa80b0540 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c004eb0 con 0x7f8fa80b0540 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493999478 0 0) 0x7f8fa80b1640 con 0x7f8fa809ac40 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa80afeb0 con 0x7f8fa809ac40 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853423911 0 0) 0x7f8fa80b0ae0 con 0x7f8fa81d7b50 2024-03-29T10:38:56.406 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa80b1640 con 0x7f8fa81d7b50 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149898131 0 0) 0x7f8f8c004eb0 con 0x7f8fa80b0540 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa80b0ae0 con 0x7f8fa80b0540 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8f98001a10 con 0x7f8fa80b0540 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): dump: 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645652929 0 0) 0x7f8fa80afeb0 con 0x7f8fa809ac40 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f8c004eb0 con 0x7f8fa809ac40 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8fa4001b00 con 0x7f8fa809ac40 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): dump: 2024-03-29T10:38:56.408 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1690645417 0 0) 0x7f8fa80b1640 con 0x7f8fa81d7b50 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa80afeb0 con 0x7f8fa81d7b50 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8fa0001bc0 con 0x7f8fa81d7b50 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient(hunting): dump: 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1891668661 0 0) 0x7f8fa80b0ae0 con 0x7f8fa80b0540 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 >> v1:172.21.15.94:6790/0 conn(0x7f8fa809ac40 legacy=0x7f8fa80af7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 >> v1:172.21.15.94:6789/0 conn(0x7f8fa81d7b50 legacy=0x7f8fa81e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.410 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 monclient: found mon.b 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa80b18b0 con 0x7f8fa80b0540 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 10 monclient: _renew_subs 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa81d8000 con 0x7f8fa80b0540 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f98001c50 con 0x7f8fa80b0540 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8f98003a60 con 0x7f8fa80b0540 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 monclient: dump: 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.412 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 10 monclient: _renew_subs 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9ffff640 4 set_mon_vals no callback set 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa81e35f0 con 0x7f8fa80b0540 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8fb0206640 1 librados: init done 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8f98004010 con 0x7f8fa80b0540 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2470118457 0 0) 0x7f8f980066f0 con 0x7f8fa80b0540 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 10 client.4111.objecter ms_dispatch 0x7f8fa8000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.408+0000 7f8f9e7fc640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.409+0000 7f8fb0206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.409+0000 7f8fb0206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.409+0000 7f8fb0206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.409+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa81d7160 con 0x7f8fa80b0540 2024-03-29T10:38:56.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.411+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8fa81d7160 con 0x7f8fa80b0540 2024-03-29T10:38:56.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.411+0000 7f8f9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.411+0000 7f8f9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:56.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.483+0000 7f8fb0206640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a30ab5d7-7189-4c97-81b1-72012cadfce5", "id": 0}] 2024-03-29T10:38:56.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.483+0000 7f8fb0206640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a30ab5d7-7189-4c97-81b1-72012cadfce5", "id": 0}] 2024-03-29T10:38:56.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.483+0000 7f8fb0206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.483+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a30ab5d7-7189-4c97-81b1-72012cadfce5", "id": 0} v 0) v1 -- 0x7f8fa809b020 con 0x7f8fa80b0540 2024-03-29T10:38:56.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.489+0000 7f8f9e7fc640 1 -- 172.21.15.173:0/2802710639 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a30ab5d7-7189-4c97-81b1-72012cadfce5", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 324868053 0 732707515) 0x7f8f98005c10 con 0x7f8fa80b0540 2024-03-29T10:38:56.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.489+0000 7f8f9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a30ab5d7-7189-4c97-81b1-72012cadfce5", "id": 0}] 2024-03-29T10:38:56.473 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.489+0000 7f8f9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:56.474 INFO:teuthology.orchestra.run.smithi173.stdout:0 2024-03-29T10:38:56.476 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 10 librados: watch_flush enter 2024-03-29T10:38:56.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 10 librados: watch_flush exit 2024-03-29T10:38:56.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:38:56.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:38:56.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 10 monclient: shutdown 2024-03-29T10:38:56.477 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 >> v1:172.21.15.173:6789/0 conn(0x7f8fa80b0540 legacy=0x7f8fa81d7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 shutdown_connections 2024-03-29T10:38:56.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 >> 172.21.15.173:0/2802710639 conn(0x7f8fa8066b00 msgr2=0x7f8fa80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 shutdown_connections 2024-03-29T10:38:56.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 1 -- 172.21.15.173:0/2802710639 wait complete. 2024-03-29T10:38:56.478 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.490+0000 7f8fb0206640 1 librados: shutdown 2024-03-29T10:38:56.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new b6136696-9783-4145-bdff-58e1bc4e42cb 1 2024-03-29T10:38:56.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.600+0000 7efc56450640 10 monclient: get_monmap_and_config 2024-03-29T10:38:56.595 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.600+0000 7efc56450640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.600+0000 7efc56450640 10 monclient: monmap: 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:56.601321+0000 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:56.601321+0000 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 1 Processor -- start 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 1 -- start start 2024-03-29T10:38:56.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient: init 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient(hunting): picked mon.noname-a con 0x7efc5009ac40 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient(hunting): picked mon.noname-b con 0x7efc5009c870 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient(hunting): picked mon.noname-c con 0x7efc500b4360 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5009dec0 con 0x7efc5009ac40 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5009da30 con 0x7efc500b4360 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500b39c0 con 0x7efc5009c870 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc56450640 10 monclient(hunting): authenticate will time out at 783.851581s 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc549ed640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7efc500b4360 0x7efc5014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34244/0 (socket says 172.21.15.173:34244) 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.601+0000 7efc549ed640 1 -- 172.21.15.173:0/2631098657 learned_addr learned my addr 172.21.15.173:0/2631098657 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.597 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038779826 0 0) 0x7efc5009da30 con 0x7efc500b4360 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc38003520 con 0x7efc500b4360 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3167121275 0 0) 0x7efc38003520 con 0x7efc500b4360 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 >> v1:172.21.15.94:6790/0 conn(0x7efc5009c870 legacy=0x7efc5013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 >> v1:172.21.15.94:6789/0 conn(0x7efc5009ac40 legacy=0x7efc500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 monclient: found mon.noname-c 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500b5160 con 0x7efc500b4360 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.598 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7efc48001a10 con 0x7efc500b4360 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: dump: 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.599 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc48001bf0 con 0x7efc500b4360 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 1 -- 172.21.15.173:0/2631098657 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7efc48002000 con 0x7efc500b4360 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 4 set_mon_vals no callback set 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc4effd640 10 monclient: dump: 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.600 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 10 monclient: shutdown 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.602+0000 7efc56450640 1 -- 172.21.15.173:0/2631098657 >> v1:172.21.15.173:6789/0 conn(0x7efc500b4360 legacy=0x7efc5014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 1 -- 172.21.15.173:0/2631098657 shutdown_connections 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 1 -- 172.21.15.173:0/2631098657 >> 172.21.15.173:0/2631098657 conn(0x7efc50066b00 msgr2=0x7efc50066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 1 -- 172.21.15.173:0/2631098657 shutdown_connections 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 1 -- 172.21.15.173:0/2631098657 wait complete. 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.601 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 10 monclient: monmap: 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 1 librados: starting msgr at 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 1 librados: starting objecter 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.603+0000 7efc56450640 1 Processor -- start 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 1 -- start start 2024-03-29T10:38:56.602 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 1 librados: setting wanted keys 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 1 librados: calling monclient init 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient: init 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient(hunting): picked mon.noname-b con 0x7efc5009ac40 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient(hunting): picked mon.noname-a con 0x7efc500afa20 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient(hunting): picked mon.noname-c con 0x7efc50141740 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc501d70d0 con 0x7efc500afa20 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5013ffa0 con 0x7efc5009ac40 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc501d7c70 con 0x7efc50141740 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc56450640 10 monclient(hunting): authenticate will time out at 783.854578s 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4ffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7efc5009ac40 0x7efc501d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34256/0 (socket says 172.21.15.173:34256) 2024-03-29T10:38:56.603 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4ffff640 1 -- 172.21.15.173:0/1918456661 learned_addr learned my addr 172.21.15.173:0/1918456661 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4cff9640 10 client.?.objecter ms_handle_connect 0x7efc5009ac40 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4cff9640 10 client.?.objecter ms_handle_connect 0x7efc500afa20 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4cff9640 10 client.?.objecter ms_handle_connect 0x7efc50141740 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.604+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838736406 0 0) 0x7efc5013ffa0 con 0x7efc5009ac40 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c004eb0 con 0x7efc5009ac40 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660014977 0 0) 0x7efc501d7c70 con 0x7efc50141740 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc5013ffa0 con 0x7efc50141740 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144661255 0 0) 0x7efc501d70d0 con 0x7efc500afa20 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc501d7c70 con 0x7efc500afa20 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080371586 0 0) 0x7efc3c004eb0 con 0x7efc5009ac40 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc501d70d0 con 0x7efc5009ac40 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7efc40001830 con 0x7efc5009ac40 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:56.605 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): dump: 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3584634175 0 0) 0x7efc501d7c70 con 0x7efc500afa20 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc3c004eb0 con 0x7efc500afa20 2024-03-29T10:38:56.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3274530709 0 0) 0x7efc5013ffa0 con 0x7efc50141740 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc501d7c70 con 0x7efc50141740 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7efc44001730 con 0x7efc500afa20 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): dump: 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.607 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7efc48001f00 con 0x7efc50141740 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient(hunting): dump: 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.608 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3432067723 0 0) 0x7efc501d70d0 con 0x7efc5009ac40 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 >> v1:172.21.15.94:6790/0 conn(0x7efc50141740 legacy=0x7efc501e1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 >> v1:172.21.15.94:6789/0 conn(0x7efc500afa20 legacy=0x7efc50141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 monclient: found mon.b 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc501d7e00 con 0x7efc5009ac40 2024-03-29T10:38:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 10 monclient: _renew_subs 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc501e3150 con 0x7efc5009ac40 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 10 monclient: _renew_subs 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc501e34c0 con 0x7efc5009ac40 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc56450640 1 librados: init done 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc40003610 con 0x7efc5009ac40 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7efc400039a0 con 0x7efc5009ac40 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4e7fc640 4 set_mon_vals no callback set 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 monclient: dump: 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.611 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efc40003f50 con 0x7efc5009ac40 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1507525439 0 0) 0x7efc40006570 con 0x7efc5009ac40 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 10 client.4117.objecter ms_dispatch 0x7efc50000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.605+0000 7efc4cff9640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.606+0000 7efc56450640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.606+0000 7efc56450640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.606+0000 7efc56450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.606+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc30005080 con 0x7efc5009ac40 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.608+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efc400057c0 con 0x7efc5009ac40 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.608+0000 7efc4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.608+0000 7efc4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:56.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.680+0000 7efc56450640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b6136696-9783-4145-bdff-58e1bc4e42cb", "id": 1}] 2024-03-29T10:38:56.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.680+0000 7efc56450640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b6136696-9783-4145-bdff-58e1bc4e42cb", "id": 1}] 2024-03-29T10:38:56.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.680+0000 7efc56450640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.680+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b6136696-9783-4145-bdff-58e1bc4e42cb", "id": 1} v 0) v1 -- 0x7efc300058b0 con 0x7efc5009ac40 2024-03-29T10:38:56.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.688+0000 7efc4cff9640 1 -- 172.21.15.173:0/1918456661 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6136696-9783-4145-bdff-58e1bc4e42cb", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 527868352 0 940483276) 0x7efc400043e0 con 0x7efc5009ac40 2024-03-29T10:38:56.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.688+0000 7efc4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b6136696-9783-4145-bdff-58e1bc4e42cb", "id": 1}] 2024-03-29T10:38:56.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.688+0000 7efc4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:56.673 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-29T10:38:56.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 10 librados: watch_flush enter 2024-03-29T10:38:56.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 10 librados: watch_flush exit 2024-03-29T10:38:56.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 10 monclient: shutdown 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 >> v1:172.21.15.173:6789/0 conn(0x7efc5009ac40 legacy=0x7efc501d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 shutdown_connections 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 >> 172.21.15.173:0/1918456661 conn(0x7efc50066b00 msgr2=0x7efc5013a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 shutdown_connections 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 1 -- 172.21.15.173:0/1918456661 wait complete. 2024-03-29T10:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.689+0000 7efc56450640 1 librados: shutdown 2024-03-29T10:38:56.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 6c79b11a-25e7-429a-a953-54e94244c8fc 2 2024-03-29T10:38:56.770 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:56.786+0000 7f6834a13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:38:56.780 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:56.796+0000 7fdf2ea20040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:38:56.783 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.799+0000 7f396368d640 10 monclient: get_monmap_and_config 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.799+0000 7f396368d640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.799+0000 7f396368d640 10 monclient: monmap: 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:56.800757+0000 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:56.800757+0000 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:56.784 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.800+0000 7f396368d640 1 Processor -- start 2024-03-29T10:38:56.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.800+0000 7f396368d640 1 -- start start 2024-03-29T10:38:56.795 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.800+0000 7f396368d640 10 monclient: init 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient(hunting): picked mon.noname-b con 0x7f395c09ac40 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient(hunting): picked mon.noname-a con 0x7f395c09c870 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient(hunting): picked mon.noname-c con 0x7f395c0b4360 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c09dec0 con 0x7f395c09c870 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c09da30 con 0x7f395c0b4360 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c0b39c0 con 0x7f395c09ac40 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient(hunting): authenticate will time out at 784.050978s 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3961c2a640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f395c0b4360 0x7f395c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34268/0 (socket says 172.21.15.173:34268) 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3961c2a640 1 -- 172.21.15.173:0/2478013577 learned_addr learned my addr 172.21.15.173:0/2478013577 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484792628 0 0) 0x7f395c09da30 con 0x7f395c0b4360 2024-03-29T10:38:56.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3948003520 con 0x7f395c0b4360 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723100065 0 0) 0x7f394c001110 con 0x7f395c09c870 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f395c09da30 con 0x7f395c09c870 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174295168 0 0) 0x7f395c0b39c0 con 0x7f395c09ac40 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c001110 con 0x7f395c09ac40 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1281231577 0 0) 0x7f395c09da30 con 0x7f395c09c870 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 >> v1:172.21.15.94:6790/0 conn(0x7f395c09ac40 legacy=0x7f395c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 >> v1:172.21.15.173:6789/0 conn(0x7f395c0b4360 legacy=0x7f395c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 monclient: found mon.noname-a 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c0b5160 con 0x7f395c09c870 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f394c001660 con 0x7f395c09c870 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 5 monclient: authenticate success, global_id 4125 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: dump: 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.799 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f394c0017c0 con 0x7f395c09c870 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f396368d640 4 set_mon_vals no callback set 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 1 -- 172.21.15.173:0/2478013577 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f394c001c10 con 0x7f395c09c870 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.801+0000 7f3953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f3953fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f3953fff640 10 monclient: dump: 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.800 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f396368d640 10 monclient: shutdown 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f396368d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f396368d640 1 -- 172.21.15.173:0/2478013577 >> v1:172.21.15.94:6789/0 conn(0x7f395c09c870 legacy=0x7f395c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f396368d640 1 -- 172.21.15.173:0/2478013577 shutdown_connections 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f396368d640 1 -- 172.21.15.173:0/2478013577 >> 172.21.15.173:0/2478013577 conn(0x7f395c066b00 msgr2=0x7f395c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f396368d640 1 -- 172.21.15.173:0/2478013577 shutdown_connections 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.802+0000 7f396368d640 1 -- 172.21.15.173:0/2478013577 wait complete. 2024-03-29T10:38:56.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 10 monclient: monmap: 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 1 librados: starting msgr at 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 1 librados: starting objecter 2024-03-29T10:38:56.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 1 Processor -- start 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 1 -- start start 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 1 librados: setting wanted keys 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 1 librados: calling monclient init 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.803+0000 7f396368d640 10 monclient: init 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 10 monclient(hunting): picked mon.noname-a con 0x7f395c09ac40 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 10 monclient(hunting): picked mon.noname-c con 0x7f395c139b60 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 10 monclient(hunting): picked mon.noname-b con 0x7f395c1461a0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c1402f0 con 0x7f395c09ac40 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c1d7630 con 0x7f395c1461a0 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c13ffa0 con 0x7f395c139b60 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f396368d640 10 monclient(hunting): authenticate will time out at 784.054115s 2024-03-29T10:38:56.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3961c2a640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f395c1461a0 0x7f395c1e24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34280/0 (socket says 172.21.15.173:34280) 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3961c2a640 1 -- 172.21.15.173:0/4264764692 learned_addr learned my addr 172.21.15.173:0/4264764692 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 client.?.objecter ms_handle_connect 0x7f395c1461a0 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 client.?.objecter ms_handle_connect 0x7f395c09ac40 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 client.?.objecter ms_handle_connect 0x7f395c139b60 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813990871 0 0) 0x7f395c1d7630 con 0x7f395c1461a0 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940004eb0 con 0x7f395c1461a0 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595131008 0 0) 0x7f395c13ffa0 con 0x7f395c139b60 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f395c1d7630 con 0x7f395c139b60 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655479148 0 0) 0x7f395c1402f0 con 0x7f395c09ac40 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:38:56.805 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f395c13ffa0 con 0x7f395c09ac40 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184467539 0 0) 0x7f395c1d7630 con 0x7f395c139b60 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f395c1402f0 con 0x7f395c139b60 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f394c001b90 con 0x7f395c139b60 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): dump: 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.806 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2080656082 0 0) 0x7f3940004eb0 con 0x7f395c1461a0 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f395c1d7630 con 0x7f395c1461a0 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4063427612 0 0) 0x7f395c13ffa0 con 0x7f395c09ac40 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3940004eb0 con 0x7f395c09ac40 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3954001b50 con 0x7f395c1461a0 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): dump: 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.807 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3958001870 con 0x7f395c09ac40 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.804+0000 7f3951ffb640 10 monclient(hunting): dump: 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.808 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4170300214 0 0) 0x7f395c1402f0 con 0x7f395c139b60 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 >> v1:172.21.15.173:6789/0 conn(0x7f395c1461a0 legacy=0x7f395c1e24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 >> v1:172.21.15.94:6789/0 conn(0x7f395c09ac40 legacy=0x7f395c145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 monclient: found mon.c 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c0b0070 con 0x7f395c139b60 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 5 monclient: authenticate success, global_id 4130 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 10 monclient: _renew_subs 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c0b0320 con 0x7f395c139b60 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f394c0038a0 con 0x7f395c139b60 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f394c003bf0 con 0x7f395c139b60 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 monclient: dump: 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f39537fe640 4 set_mon_vals no callback set 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 10 monclient: _renew_subs 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f394c004160 con 0x7f395c139b60 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c1d7080 con 0x7f395c139b60 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f396368d640 1 librados: init done 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2639223479 0 0) 0x7f394c004830 con 0x7f395c139b60 2024-03-29T10:38:56.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 10 client.4130.objecter ms_dispatch 0x7f395c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 3 client.4130.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 3 client.4130.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.805+0000 7f3951ffb640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.806+0000 7f396368d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.806+0000 7f396368d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.806+0000 7f396368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.806+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c1465b0 con 0x7f395c139b60 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.808+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f394c004410 con 0x7f395c139b60 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.808+0000 7f3951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.808+0000 7f3951ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:56.824 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:56.840+0000 7f6834a13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:38:56.837 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:56.853+0000 7fdf2ea20040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:38:56.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.880+0000 7f396368d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6c79b11a-25e7-429a-a953-54e94244c8fc", "id": 2}] 2024-03-29T10:38:56.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.880+0000 7f396368d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6c79b11a-25e7-429a-a953-54e94244c8fc", "id": 2}] 2024-03-29T10:38:56.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.880+0000 7f396368d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:56.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.880+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd new", "uuid": "6c79b11a-25e7-429a-a953-54e94244c8fc", "id": 2} v 0) v1 -- 0x7f395c074bb0 con 0x7f395c139b60 2024-03-29T10:38:56.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.883+0000 7f3951ffb640 1 -- 172.21.15.173:0/4264764692 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c79b11a-25e7-429a-a953-54e94244c8fc", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3607523301 0 216599125) 0x7f395c074bb0 con 0x7f395c139b60 2024-03-29T10:38:56.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.883+0000 7f3951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6c79b11a-25e7-429a-a953-54e94244c8fc", "id": 2}] 2024-03-29T10:38:56.868 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.883+0000 7f3951ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:56.868 INFO:teuthology.orchestra.run.smithi173.stdout:2 2024-03-29T10:38:56.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 10 librados: watch_flush enter 2024-03-29T10:38:56.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 10 librados: watch_flush exit 2024-03-29T10:38:56.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-29T10:38:56.869 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 10 client.4130.objecter successfully canceled tick 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 10 monclient: shutdown 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 >> v1:172.21.15.94:6790/0 conn(0x7f395c139b60 legacy=0x7f395c0af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 shutdown_connections 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 >> 172.21.15.173:0/4264764692 conn(0x7f395c066b00 msgr2=0x7f395c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 shutdown_connections 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 1 -- 172.21.15.173:0/4264764692 wait complete. 2024-03-29T10:38:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.884+0000 7f396368d640 1 librados: shutdown 2024-03-29T10:38:56.871 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:56.885+0000 7f6834a13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:38:56.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 8b191e31-05ef-44ed-ae87-a385f46165f5 3 2024-03-29T10:38:56.884 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:56.900+0000 7fdf2ea20040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:38:56.923 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:56.939+0000 7f6834a13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:38:56.941 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:56.957+0000 7fdf2ea20040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:38:56.964 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:56.980+0000 7f6834a13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.995+0000 7f64f8fc1640 10 monclient: get_monmap_and_config 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.995+0000 7f64f8fc1640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.995+0000 7f64f8fc1640 10 monclient: monmap: 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:56.996599+0000 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:56.996599+0000 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:56.980 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.981 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 1 Processor -- start 2024-03-29T10:38:56.981 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 1 -- start start 2024-03-29T10:38:56.981 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient: init 2024-03-29T10:38:56.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:56.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient(hunting): picked mon.noname-a con 0x7f64f409ac40 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient(hunting): picked mon.noname-c con 0x7f64f409c870 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.989 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient(hunting): picked mon.noname-b con 0x7f64f40b4360 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f409dec0 con 0x7f64f409ac40 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f409da30 con 0x7f64f409c870 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f40b39c0 con 0x7f64f40b4360 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.996+0000 7f64f8fc1640 10 monclient(hunting): authenticate will time out at 784.246832s 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f1d9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f64f409c870 0x7f64f413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34290/0 (socket says 172.21.15.173:34290) 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f1d9b640 1 -- 172.21.15.173:0/2896477583 learned_addr learned my addr 172.21.15.173:0/2896477583 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266949754 0 0) 0x7f64f409da30 con 0x7f64f409c870 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64dc003520 con 0x7f64f409c870 2024-03-29T10:38:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506497627 0 0) 0x7f64e4001110 con 0x7f64f409ac40 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f409da30 con 0x7f64f409ac40 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115315316 0 0) 0x7f64f40b39c0 con 0x7f64f40b4360 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e4001110 con 0x7f64f40b4360 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2185542083 0 0) 0x7f64dc003520 con 0x7f64f409c870 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 >> v1:172.21.15.94:6790/0 conn(0x7f64f40b4360 legacy=0x7f64f414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 >> v1:172.21.15.94:6789/0 conn(0x7f64f409ac40 legacy=0x7f64f40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.991 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 monclient: found mon.noname-c 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f40b5160 con 0x7f64f409c870 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64e8001610 con 0x7f64f409c870 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f8fc1640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f8fc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: dump: 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.992 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f8fc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64e8001770 con 0x7f64f409c870 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 1 -- 172.21.15.173:0/2896477583 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64e8001bc0 con 0x7f64f409c870 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.993 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f159a640 10 monclient: dump: 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f8fc1640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.997+0000 7f64f8fc1640 4 set_mon_vals no callback set 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.998+0000 7f64f8fc1640 10 monclient: shutdown 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.998+0000 7f64f8fc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.998+0000 7f64f8fc1640 1 -- 172.21.15.173:0/2896477583 >> v1:172.21.15.173:6789/0 conn(0x7f64f409c870 legacy=0x7f64f413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.998+0000 7f64f8fc1640 1 -- 172.21.15.173:0/2896477583 shutdown_connections 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.998+0000 7f64f8fc1640 1 -- 172.21.15.173:0/2896477583 >> 172.21.15.173:0/2896477583 conn(0x7f64f4066b00 msgr2=0x7f64f4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.998+0000 7f64f8fc1640 1 -- 172.21.15.173:0/2896477583 shutdown_connections 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.998+0000 7f64f8fc1640 1 -- 172.21.15.173:0/2896477583 wait complete. 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.999+0000 7f64f8fc1640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:56.999+0000 7f64f8fc1640 10 monclient: monmap: 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:56.995 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:56.996 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:56.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.000+0000 7f64f8fc1640 1 librados: starting msgr at 2024-03-29T10:38:56.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.000+0000 7f64f8fc1640 1 librados: starting objecter 2024-03-29T10:38:56.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.000+0000 7f64f8fc1640 1 Processor -- start 2024-03-29T10:38:56.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.000+0000 7f64f8fc1640 1 -- start start 2024-03-29T10:38:56.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.000+0000 7f64f8fc1640 1 librados: setting wanted keys 2024-03-29T10:38:56.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.000+0000 7f64f8fc1640 1 librados: calling monclient init 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.000+0000 7f64f8fc1640 10 monclient: init 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 10 monclient(hunting): picked mon.noname-c con 0x7f64f409ac40 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 10 monclient(hunting): picked mon.noname-b con 0x7f64f4139080 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 10 monclient(hunting): picked mon.noname-a con 0x7f64f41d7420 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f41d7fc0 con 0x7f64f41d7420 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f40b19b0 con 0x7f64f4139080 2024-03-29T10:38:56.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f4139d90 con 0x7f64f409ac40 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f8fc1640 10 monclient(hunting): authenticate will time out at 784.251128s 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f1d9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f64f4139080 0x7f64f4139530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34306/0 (socket says 172.21.15.173:34306) 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64f1d9b640 1 -- 172.21.15.173:0/369384707 learned_addr learned my addr 172.21.15.173:0/369384707 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 client.?.objecter ms_handle_connect 0x7f64f4139080 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 client.?.objecter ms_handle_connect 0x7f64f409ac40 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 client.?.objecter ms_handle_connect 0x7f64f41d7420 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278325893 0 0) 0x7f64f40b19b0 con 0x7f64f4139080 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d4004eb0 con 0x7f64f4139080 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036223297 0 0) 0x7f64f41d7fc0 con 0x7f64f41d7420 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f40b19b0 con 0x7f64f41d7420 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971671441 0 0) 0x7f64f4139d90 con 0x7f64f409ac40 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f41d7fc0 con 0x7f64f409ac40 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2156559562 0 0) 0x7f64d4004eb0 con 0x7f64f4139080 2024-03-29T10:38:57.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64f4139d90 con 0x7f64f4139080 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64e8001b40 con 0x7f64f4139080 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): dump: 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.001 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 372496717 0 0) 0x7f64f40b19b0 con 0x7f64f41d7420 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64d4004eb0 con 0x7f64f41d7420 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3097904079 0 0) 0x7f64f41d7fc0 con 0x7f64f409ac40 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64f40b19b0 con 0x7f64f409ac40 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64ec001a70 con 0x7f64f41d7420 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): dump: 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64e4001860 con 0x7f64f409ac40 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.001+0000 7f64e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient(hunting): dump: 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2072486222 0 0) 0x7f64f4139d90 con 0x7f64f4139080 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 >> v1:172.21.15.94:6790/0 conn(0x7f64f409ac40 legacy=0x7f64f40b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 >> v1:172.21.15.94:6789/0 conn(0x7f64f41d7420 legacy=0x7f64f41d78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 monclient: found mon.b 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f40b0b10 con 0x7f64f4139080 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 10 monclient: _renew_subs 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f40af630 con 0x7f64f4139080 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 10 monclient: _renew_subs 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f41e3b90 con 0x7f64f4139080 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64e8004a90 con 0x7f64f4139080 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f8fc1640 1 librados: init done 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64e8004da0 con 0x7f64f4139080 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 monclient: dump: 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.006 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64f0d99640 4 set_mon_vals no callback set 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f64e8005290 con 0x7f64f4139080 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2770238572 0 0) 0x7f64e80059d0 con 0x7f64f4139080 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 10 client.4129.objecter ms_dispatch 0x7f64f4000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.002+0000 7f64e2ffd640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.003+0000 7f64f8fc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.003+0000 7f64f8fc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.003+0000 7f64f8fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.007 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.003+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f409b020 con 0x7f64f4139080 2024-03-29T10:38:57.008 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.004+0000 7fdf2ea20040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:38:57.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.004+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f64e8005cf0 con 0x7f64f4139080 2024-03-29T10:38:57.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.004+0000 7f64e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.008 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.004+0000 7f64e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:57.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.075+0000 7f64f8fc1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8b191e31-05ef-44ed-ae87-a385f46165f5", "id": 3}] 2024-03-29T10:38:57.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.075+0000 7f64f8fc1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8b191e31-05ef-44ed-ae87-a385f46165f5", "id": 3}] 2024-03-29T10:38:57.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.075+0000 7f64f8fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.059 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.075+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8b191e31-05ef-44ed-ae87-a385f46165f5", "id": 3} v 0) v1 -- 0x7f64f404f730 con 0x7f64f4139080 2024-03-29T10:38:57.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.080+0000 7f64e2ffd640 1 -- 172.21.15.173:0/369384707 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8b191e31-05ef-44ed-ae87-a385f46165f5", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1980684009 0 525046306) 0x7f64e8005610 con 0x7f64f4139080 2024-03-29T10:38:57.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.080+0000 7f64e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8b191e31-05ef-44ed-ae87-a385f46165f5", "id": 3}] 2024-03-29T10:38:57.064 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.080+0000 7f64e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:57.064 INFO:teuthology.orchestra.run.smithi173.stdout:3 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 10 librados: watch_flush enter 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 10 librados: watch_flush exit 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 10 monclient: shutdown 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 >> v1:172.21.15.173:6789/0 conn(0x7f64f4139080 legacy=0x7f64f4139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 shutdown_connections 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 >> 172.21.15.173:0/369384707 conn(0x7f64f4066b00 msgr2=0x7f64f413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 shutdown_connections 2024-03-29T10:38:57.066 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 1 -- 172.21.15.173:0/369384707 wait complete. 2024-03-29T10:38:57.067 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.081+0000 7f64f8fc1640 1 librados: shutdown 2024-03-29T10:38:57.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 0ae457fd-cdc7-4cad-872a-edf9e37c4a6f 4 2024-03-29T10:38:57.080 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.096+0000 7f6834a13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:38:57.107 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.123+0000 7fdf2ea20040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:38:57.148 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.164+0000 7f6834a13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:38:57.181 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.196+0000 7fdf2ea20040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:38:57.193 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.210+0000 7f6834a13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:38:57.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.222+0000 7fe062990640 10 monclient: get_monmap_and_config 2024-03-29T10:38:57.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.222+0000 7fe062990640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.206 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.222+0000 7fe062990640 10 monclient: monmap: 2024-03-29T10:38:57.206 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.206 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.206 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:57.223394+0000 2024-03-29T10:38:57.206 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:57.223394+0000 2024-03-29T10:38:57.207 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.207 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.207 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.207 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:57.207 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:57.207 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 1 Processor -- start 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 1 -- start start 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient: init 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c09ac40 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c09c870 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c0b4360 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c09dec0 con 0x7fe05c09c870 2024-03-29T10:38:57.217 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c09da30 con 0x7fe05c0b4360 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c0b39c0 con 0x7fe05c09ac40 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe062990640 10 monclient(hunting): authenticate will time out at 784.473705s 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe060f2d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe05c0b4360 0x7fe05c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34314/0 (socket says 172.21.15.173:34314) 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.223+0000 7fe060f2d640 1 -- 172.21.15.173:0/1637864914 learned_addr learned my addr 172.21.15.173:0/1637864914 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778177435 0 0) 0x7fe05c09da30 con 0x7fe05c0b4360 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe044003520 con 0x7fe05c0b4360 2024-03-29T10:38:57.218 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824090654 0 0) 0x7fe050001110 con 0x7fe05c09c870 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c09da30 con 0x7fe05c09c870 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264469689 0 0) 0x7fe05c0b39c0 con 0x7fe05c09ac40 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050001110 con 0x7fe05c09ac40 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523534095 0 0) 0x7fe044003520 con 0x7fe05c0b4360 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 >> v1:172.21.15.94:6790/0 conn(0x7fe05c09ac40 legacy=0x7fe05c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.219 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 >> v1:172.21.15.94:6789/0 conn(0x7fe05c09c870 legacy=0x7fe05c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 monclient: found mon.noname-c 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0b5160 con 0x7fe05c0b4360 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe054001a10 con 0x7fe05c0b4360 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: dump: 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.220 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe062990640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe062990640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe054001b70 con 0x7fe05c0b4360 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 1 -- 172.21.15.173:0/1637864914 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe054001fc0 con 0x7fe05c0b4360 2024-03-29T10:38:57.221 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe05affd640 10 monclient: dump: 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe062990640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:57.222 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.224+0000 7fe062990640 4 set_mon_vals no callback set 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 10 monclient: shutdown 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 1 -- 172.21.15.173:0/1637864914 >> v1:172.21.15.173:6789/0 conn(0x7fe05c0b4360 legacy=0x7fe05c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 1 -- 172.21.15.173:0/1637864914 shutdown_connections 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 1 -- 172.21.15.173:0/1637864914 >> 172.21.15.173:0/1637864914 conn(0x7fe05c066b00 msgr2=0x7fe05c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 1 -- 172.21.15.173:0/1637864914 shutdown_connections 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 1 -- 172.21.15.173:0/1637864914 wait complete. 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.225+0000 7fe062990640 10 monclient: monmap: 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:57.223 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 1 librados: starting msgr at 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 1 librados: starting objecter 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 1 Processor -- start 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 1 -- start start 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 1 librados: setting wanted keys 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 1 librados: calling monclient init 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 10 monclient: init 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.226+0000 7fe062990640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:57.224 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c09ac40 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c0b10f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c1461f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c1430b0 con 0x7fe05c09ac40 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c142940 con 0x7fe05c1461f0 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c142ad0 con 0x7fe05c0b10f0 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient(hunting): authenticate will time out at 784.476957s 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe060f2d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe05c1461f0 0x7fe05c1e1810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34326/0 (socket says 172.21.15.173:34326) 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe060f2d640 1 -- 172.21.15.173:0/1644270533 learned_addr learned my addr 172.21.15.173:0/1644270533 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 client.?.objecter ms_handle_connect 0x7fe05c1461f0 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 client.?.objecter ms_handle_connect 0x7fe05c0b10f0 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 client.?.objecter ms_handle_connect 0x7fe05c09ac40 2024-03-29T10:38:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633454202 0 0) 0x7fe05c142940 con 0x7fe05c1461f0 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe048004eb0 con 0x7fe05c1461f0 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882215190 0 0) 0x7fe05c1430b0 con 0x7fe05c09ac40 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c142940 con 0x7fe05c09ac40 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087434174 0 0) 0x7fe05c142ad0 con 0x7fe05c0b10f0 2024-03-29T10:38:57.226 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c1430b0 con 0x7fe05c0b10f0 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250347446 0 0) 0x7fe05c142940 con 0x7fe05c09ac40 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe05c142ad0 con 0x7fe05c09ac40 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe04c001910 con 0x7fe05c09ac40 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): dump: 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.227 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1657618662 0 0) 0x7fe048004eb0 con 0x7fe05c1461f0 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe05c142940 con 0x7fe05c1461f0 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe054001ec0 con 0x7fe05c1461f0 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): dump: 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.228 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293909787 0 0) 0x7fe05c1430b0 con 0x7fe05c0b10f0 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe048004eb0 con 0x7fe05c0b10f0 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe0500018a0 con 0x7fe05c0b10f0 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:57.229 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient(hunting): dump: 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2769657213 0 0) 0x7fe05c142ad0 con 0x7fe05c09ac40 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 >> v1:172.21.15.94:6790/0 conn(0x7fe05c0b10f0 legacy=0x7fe05c1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.230 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 >> v1:172.21.15.173:6789/0 conn(0x7fe05c1461f0 legacy=0x7fe05c1e1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 monclient: found mon.a 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c142c60 con 0x7fe05c09ac40 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe058ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 5 monclient: authenticate success, global_id 4140 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient: _renew_subs 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c1e2f30 con 0x7fe05c09ac40 2024-03-29T10:38:57.231 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient: _renew_subs 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.227+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c1e32a0 con 0x7fe05c09ac40 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe062990640 1 librados: init done 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe04c001490 con 0x7fe05c09ac40 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe04c003790 con 0x7fe05c09ac40 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe05a7fc640 4 set_mon_vals no callback set 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 10 monclient: dump: 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.232 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe04c003c80 con 0x7fe05c09ac40 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2225839089 0 0) 0x7fe04c0053c0 con 0x7fe05c09ac40 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 10 client.4140.objecter ms_dispatch 0x7fe05c000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.228+0000 7fe058ff9640 20 client.4140.objecter dump_active .. 0 homeless 2024-03-29T10:38:57.233 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.229+0000 7fe062990640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.229+0000 7fe062990640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.229+0000 7fe062990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.229+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03c005080 con 0x7fe05c09ac40 2024-03-29T10:38:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.231+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe04c004eb0 con 0x7fe05c09ac40 2024-03-29T10:38:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.231+0000 7fe058ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.231+0000 7fe058ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:57.235 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.245+0000 7fdf2ea20040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:38:57.246 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.262+0000 7f6834a13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:38:57.283 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.299+0000 7fdf2ea20040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:38:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.304+0000 7fe062990640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0ae457fd-cdc7-4cad-872a-edf9e37c4a6f", "id": 4}] 2024-03-29T10:38:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.304+0000 7fe062990640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0ae457fd-cdc7-4cad-872a-edf9e37c4a6f", "id": 4}] 2024-03-29T10:38:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.304+0000 7fe062990640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:57.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.304+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0ae457fd-cdc7-4cad-872a-edf9e37c4a6f", "id": 4} v 0) v1 -- 0x7fe03c0058b0 con 0x7fe05c09ac40 2024-03-29T10:38:57.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.305+0000 7fe058ff9640 1 -- 172.21.15.173:0/1644270533 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ae457fd-cdc7-4cad-872a-edf9e37c4a6f", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3105858939 0 1697012583) 0x7fe04c005600 con 0x7fe05c09ac40 2024-03-29T10:38:57.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.305+0000 7fe058ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0ae457fd-cdc7-4cad-872a-edf9e37c4a6f", "id": 4}] 2024-03-29T10:38:57.289 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.305+0000 7fe058ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:57.290 INFO:teuthology.orchestra.run.smithi173.stdout:4 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 10 librados: watch_flush enter 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 10 librados: watch_flush exit 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 20 client.4140.objecter shutdown clearing up homeless session... 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 10 client.4140.objecter successfully canceled tick 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 10 monclient: shutdown 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 >> v1:172.21.15.94:6789/0 conn(0x7fe05c09ac40 legacy=0x7fe05c1d6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 shutdown_connections 2024-03-29T10:38:57.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 >> 172.21.15.173:0/1644270533 conn(0x7fe05c066b00 msgr2=0x7fe05c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 shutdown_connections 2024-03-29T10:38:57.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 1 -- 172.21.15.173:0/1644270533 wait complete. 2024-03-29T10:38:57.292 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.306+0000 7fe062990640 1 librados: shutdown 2024-03-29T10:38:57.293 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.309+0000 7f6834a13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:38:57.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 1979d004-656d-4403-a22c-080213311414 5 2024-03-29T10:38:57.334 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.350+0000 7fdf2ea20040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:38:57.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.416+0000 7f21bac0d640 10 monclient: get_monmap_and_config 2024-03-29T10:38:57.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.416+0000 7f21bac0d640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.416+0000 7f21bac0d640 10 monclient: monmap: 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:57.417504+0000 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:57.417504+0000 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:57.401 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 1 Processor -- start 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 1 -- start start 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient: init 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient(hunting): picked mon.noname-a con 0x7f21b409ac40 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient(hunting): picked mon.noname-b con 0x7f21b409c870 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient(hunting): picked mon.noname-c con 0x7f21b40b4360 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b409dec0 con 0x7f21b409ac40 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b409da30 con 0x7f21b40b4360 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b40b39c0 con 0x7f21b409c870 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.417+0000 7f21bac0d640 10 monclient(hunting): authenticate will time out at 784.667707s 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b91aa640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f21b40b4360 0x7f21b414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34332/0 (socket says 172.21.15.173:34332) 2024-03-29T10:38:57.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b91aa640 1 -- 172.21.15.173:0/1303176129 learned_addr learned my addr 172.21.15.173:0/1303176129 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894568175 0 0) 0x7f21b409da30 con 0x7f21b40b4360 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a4003520 con 0x7f21b40b4360 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153505129 0 0) 0x7f21a8001110 con 0x7f21b409ac40 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b409da30 con 0x7f21b409ac40 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714445583 0 0) 0x7f21b40b39c0 con 0x7f21b409c870 2024-03-29T10:38:57.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a8001110 con 0x7f21b409c870 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 719018858 0 0) 0x7f21a4003520 con 0x7f21b40b4360 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 >> v1:172.21.15.94:6790/0 conn(0x7f21b409c870 legacy=0x7f21b413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 >> v1:172.21.15.94:6789/0 conn(0x7f21b409ac40 legacy=0x7f21b40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 monclient: found mon.noname-c 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b40b5160 con 0x7f21b40b4360 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f21ac001a10 con 0x7f21b40b4360 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21bac0d640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21bac0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: dump: 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.415 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21bac0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21ac001bf0 con 0x7f21b40b4360 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21bac0d640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21bac0d640 4 set_mon_vals no callback set 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 1 -- 172.21.15.173:0/1303176129 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f21ac002000 con 0x7f21b40b4360 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.418+0000 7f21b37fe640 10 monclient: dump: 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.416 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 10 monclient: shutdown 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 1 -- 172.21.15.173:0/1303176129 >> v1:172.21.15.173:6789/0 conn(0x7f21b40b4360 legacy=0x7f21b414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 1 -- 172.21.15.173:0/1303176129 shutdown_connections 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 1 -- 172.21.15.173:0/1303176129 >> 172.21.15.173:0/1303176129 conn(0x7f21b4066b00 msgr2=0x7f21b4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 1 -- 172.21.15.173:0/1303176129 shutdown_connections 2024-03-29T10:38:57.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 1 -- 172.21.15.173:0/1303176129 wait complete. 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 10 monclient: monmap: 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 1 librados: starting msgr at 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.419+0000 7f21bac0d640 1 librados: starting objecter 2024-03-29T10:38:57.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 1 Processor -- start 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 1 -- start start 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 1 librados: setting wanted keys 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 1 librados: calling monclient init 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient: init 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient(hunting): picked mon.noname-c con 0x7f21b409ac40 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient(hunting): picked mon.noname-a con 0x7f21b4142c20 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient(hunting): picked mon.noname-b con 0x7f21b41d7dc0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b41d4d30 con 0x7f21b4142c20 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b41430b0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b41d71b0 con 0x7f21b409ac40 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21bac0d640 10 monclient(hunting): authenticate will time out at 784.670704s 2024-03-29T10:38:57.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21b91aa640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f21b41d7dc0 0x7f21b4145a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34344/0 (socket says 172.21.15.173:34344) 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21b91aa640 1 -- 172.21.15.173:0/3543505514 learned_addr learned my addr 172.21.15.173:0/3543505514 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.420+0000 7f21b17fa640 10 client.?.objecter ms_handle_connect 0x7f21b41d7dc0 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 client.?.objecter ms_handle_connect 0x7f21b409ac40 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822213682 0 0) 0x7f21b41430b0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2198004eb0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 client.?.objecter ms_handle_connect 0x7f21b4142c20 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162178151 0 0) 0x7f21b41d71b0 con 0x7f21b409ac40 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b41430b0 con 0x7f21b409ac40 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627134465 0 0) 0x7f21b41d4d30 con 0x7f21b4142c20 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b41d71b0 con 0x7f21b4142c20 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2272765159 0 0) 0x7f2198004eb0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b41d4d30 con 0x7f21b41d7dc0 2024-03-29T10:38:57.421 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f21ac001f40 con 0x7f21b41d7dc0 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): dump: 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:57.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 233102038 0 0) 0x7f21b41d71b0 con 0x7f21b4142c20 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2198004eb0 con 0x7f21b4142c20 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f21a0001830 con 0x7f21b4142c20 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): dump: 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.423 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106944056 0 0) 0x7f21b41430b0 con 0x7f21b409ac40 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b41d71b0 con 0x7f21b409ac40 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f21a80018a0 con 0x7f21b409ac40 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient(hunting): dump: 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3516840686 0 0) 0x7f21b41d4d30 con 0x7f21b41d7dc0 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 >> v1:172.21.15.94:6790/0 conn(0x7f21b409ac40 legacy=0x7f21b4143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 >> v1:172.21.15.94:6789/0 conn(0x7f21b4142c20 legacy=0x7f21b41d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 monclient: found mon.b 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4144ce0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 10 monclient: _renew_subs 2024-03-29T10:38:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b4145490 con 0x7f21b41d7dc0 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 10 monclient: _renew_subs 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b4146330 con 0x7f21b41d7dc0 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21bac0d640 1 librados: init done 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21ac004e10 con 0x7f21b41d7dc0 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f21ac005120 con 0x7f21b41d7dc0 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 monclient: dump: 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.426 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b2ffd640 4 set_mon_vals no callback set 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f21ac005610 con 0x7f21b41d7dc0 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3381553964 0 0) 0x7f21ac005e40 con 0x7f21b41d7dc0 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 10 client.4141.objecter ms_dispatch 0x7f21b4000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:38:57.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.421+0000 7f21b17fa640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.422+0000 7f21bac0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.422+0000 7f21bac0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.422+0000 7f21bac0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.422+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2194005080 con 0x7f21b41d7dc0 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.424+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f21ac0060a0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.424+0000 7f21b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.424+0000 7f21b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:57.480 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.496+0000 7f21bac0d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1979d004-656d-4403-a22c-080213311414", "id": 5}] 2024-03-29T10:38:57.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.496+0000 7f21bac0d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1979d004-656d-4403-a22c-080213311414", "id": 5}] 2024-03-29T10:38:57.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.496+0000 7f21bac0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.481 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.496+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1979d004-656d-4403-a22c-080213311414", "id": 5} v 0) v1 -- 0x7f21940058b0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.500+0000 7f21b17fa640 1 -- 172.21.15.173:0/3543505514 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1979d004-656d-4403-a22c-080213311414", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2837269076 0 1988412176) 0x7f21ac0058c0 con 0x7f21b41d7dc0 2024-03-29T10:38:57.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.500+0000 7f21b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1979d004-656d-4403-a22c-080213311414", "id": 5}] 2024-03-29T10:38:57.484 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.500+0000 7f21b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:57.484 INFO:teuthology.orchestra.run.smithi173.stdout:5 2024-03-29T10:38:57.485 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 10 librados: watch_flush enter 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 10 librados: watch_flush exit 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 10 monclient: shutdown 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 >> v1:172.21.15.173:6789/0 conn(0x7f21b41d7dc0 legacy=0x7f21b4145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 shutdown_connections 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 >> 172.21.15.173:0/3543505514 conn(0x7f21b4066b00 msgr2=0x7f21b413a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 shutdown_connections 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 1 -- 172.21.15.173:0/3543505514 wait complete. 2024-03-29T10:38:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.501+0000 7f21bac0d640 1 librados: shutdown 2024-03-29T10:38:57.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new a33f62a9-e415-4be0-a553-c3c6085e1d3d 6 2024-03-29T10:38:57.638 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.654+0000 7f5a75d4b640 10 monclient: get_monmap_and_config 2024-03-29T10:38:57.638 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.654+0000 7f5a75d4b640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.654+0000 7f5a75d4b640 10 monclient: monmap: 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:57.655510+0000 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:57.655510+0000 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:57.639 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 1 Processor -- start 2024-03-29T10:38:57.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 1 -- start start 2024-03-29T10:38:57.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient: init 2024-03-29T10:38:57.653 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a700b8240 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a700b1f60 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7009c8c0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7009bd80 con 0x7f5a700b8240 2024-03-29T10:38:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7009b960 con 0x7f5a7009c8c0 2024-03-29T10:38:57.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a700b2ec0 con 0x7f5a700b1f60 2024-03-29T10:38:57.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a75d4b640 10 monclient(hunting): authenticate will time out at 784.905713s 2024-03-29T10:38:57.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a6ffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f5a7009c8c0 0x7f5a7014d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34356/0 (socket says 172.21.15.173:34356) 2024-03-29T10:38:57.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.655+0000 7f5a6ffff640 1 -- 172.21.15.173:0/238250769 learned_addr learned my addr 172.21.15.173:0/238250769 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129694794 0 0) 0x7f5a7009b960 con 0x7f5a7009c8c0 2024-03-29T10:38:57.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.656 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:38:57.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a58003520 con 0x7f5a7009c8c0 2024-03-29T10:38:57.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671326055 0 0) 0x7f5a7009bd80 con 0x7f5a700b8240 2024-03-29T10:38:57.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:38:57.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a7009b960 con 0x7f5a700b8240 2024-03-29T10:38:57.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982549694 0 0) 0x7f5a700b2ec0 con 0x7f5a700b1f60 2024-03-29T10:38:57.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:38:57.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a7009bd80 con 0x7f5a700b1f60 2024-03-29T10:38:57.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097439749 0 0) 0x7f5a58003520 con 0x7f5a7009c8c0 2024-03-29T10:38:57.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 >> v1:172.21.15.94:6790/0 conn(0x7f5a700b1f60 legacy=0x7f5a7009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 >> v1:172.21.15.94:6789/0 conn(0x7f5a700b8240 legacy=0x7f5a7013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.660 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7013fb60 con 0x7f5a7009c8c0 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5a68001a10 con 0x7f5a7009c8c0 2024-03-29T10:38:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a75d4b640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:38:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a75d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:57.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: dump: 2024-03-29T10:38:57.664 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.664 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.664 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.664 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.665 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.665 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.665 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.665 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.666 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.666 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a75d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:57.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a68001bf0 con 0x7f5a7009c8c0 2024-03-29T10:38:57.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a75d4b640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:57.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 1 -- 172.21.15.173:0/238250769 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5a68002000 con 0x7f5a7009c8c0 2024-03-29T10:38:57.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a75d4b640 4 set_mon_vals no callback set 2024-03-29T10:38:57.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a6e7fc640 10 monclient: dump: 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.668 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.656+0000 7f5a75d4b640 10 monclient: shutdown 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 -- 172.21.15.173:0/238250769 >> v1:172.21.15.173:6789/0 conn(0x7f5a7009c8c0 legacy=0x7f5a7014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 -- 172.21.15.173:0/238250769 shutdown_connections 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 -- 172.21.15.173:0/238250769 >> 172.21.15.173:0/238250769 conn(0x7f5a70066b00 msgr2=0x7f5a70066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 -- 172.21.15.173:0/238250769 shutdown_connections 2024-03-29T10:38:57.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 -- 172.21.15.173:0/238250769 wait complete. 2024-03-29T10:38:57.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 10 monclient: monmap: 2024-03-29T10:38:57.670 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.670 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:57.670 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:57.670 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 librados: starting msgr at 2024-03-29T10:38:57.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 librados: starting objecter 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.657+0000 7f5a75d4b640 1 Processor -- start 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 1 -- start start 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 1 librados: setting wanted keys 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 1 librados: calling monclient init 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient: init 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f5a7009c8c0 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f5a700b8240 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f5a70142620 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a70135830 con 0x7f5a700b8240 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a701342b0 con 0x7f5a70142620 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a70133e30 con 0x7f5a7009c8c0 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a75d4b640 10 monclient(hunting): authenticate will time out at 784.908602s 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a6ffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f5a70142620 0x7f5a701e1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34360/0 (socket says 172.21.15.173:34360) 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a6ffff640 1 -- 172.21.15.173:0/2733771455 learned_addr learned my addr 172.21.15.173:0/2733771455 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a57fff640 10 client.?.objecter ms_handle_connect 0x7f5a70142620 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a57fff640 10 client.?.objecter ms_handle_connect 0x7f5a700b8240 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a57fff640 10 client.?.objecter ms_handle_connect 0x7f5a7009c8c0 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.658+0000 7f5a57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492560082 0 0) 0x7f5a701342b0 con 0x7f5a70142620 2024-03-29T10:38:57.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c004eb0 con 0x7f5a70142620 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411628250 0 0) 0x7f5a70135830 con 0x7f5a700b8240 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a701342b0 con 0x7f5a700b8240 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770308699 0 0) 0x7f5a70133e30 con 0x7f5a7009c8c0 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70135830 con 0x7f5a7009c8c0 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189499776 0 0) 0x7f5a5c004eb0 con 0x7f5a70142620 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a70133e30 con 0x7f5a70142620 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5a68001f00 con 0x7f5a70142620 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): dump: 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795904548 0 0) 0x7f5a701342b0 con 0x7f5a700b8240 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a5c004eb0 con 0x7f5a700b8240 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5a64001870 con 0x7f5a700b8240 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): dump: 2024-03-29T10:38:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.677 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189443688 0 0) 0x7f5a70135830 con 0x7f5a7009c8c0 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a701342b0 con 0x7f5a7009c8c0 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5a60001640 con 0x7f5a7009c8c0 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient(hunting): dump: 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.678 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 407871229 0 0) 0x7f5a70133e30 con 0x7f5a70142620 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 >> v1:172.21.15.94:6790/0 conn(0x7f5a7009c8c0 legacy=0x7f5a701da640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 >> v1:172.21.15.94:6789/0 conn(0x7f5a700b8240 legacy=0x7f5a70133720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 monclient: found mon.b 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a701347d0 con 0x7f5a70142620 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 10 monclient: _renew_subs 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70134a80 con 0x7f5a70142620 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 10 monclient: _renew_subs 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a701e34e0 con 0x7f5a70142620 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a75d4b640 1 librados: init done 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a68004f70 con 0x7f5a70142620 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5a68001460 con 0x7f5a70142620 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a6dffb640 4 set_mon_vals no callback set 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 monclient: dump: 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.681 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5a68005610 con 0x7f5a70142620 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1476826435 0 0) 0x7f5a68006e30 con 0x7f5a70142620 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 10 client.4147.objecter ms_dispatch 0x7f5a70000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.659+0000 7f5a57fff640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.660+0000 7f5a75d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.660+0000 7f5a75d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.660+0000 7f5a75d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.660+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a50005080 con 0x7f5a70142620 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.662+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5a680068f0 con 0x7f5a70142620 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.662+0000 7f5a57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.682 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.662+0000 7f5a57fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:57.683 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.685+0000 7f6834a13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:38:57.718 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.734+0000 7f5a75d4b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a33f62a9-e415-4be0-a553-c3c6085e1d3d", "id": 6}] 2024-03-29T10:38:57.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.734+0000 7f5a75d4b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a33f62a9-e415-4be0-a553-c3c6085e1d3d", "id": 6}] 2024-03-29T10:38:57.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.734+0000 7f5a75d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.719 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.734+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a33f62a9-e415-4be0-a553-c3c6085e1d3d", "id": 6} v 0) v1 -- 0x7f5a500058b0 con 0x7f5a70142620 2024-03-29T10:38:57.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.738+0000 7f5a57fff640 1 -- 172.21.15.173:0/2733771455 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a33f62a9-e415-4be0-a553-c3c6085e1d3d", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3460977626 0 1113811849) 0x7f5a68006b60 con 0x7f5a70142620 2024-03-29T10:38:57.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.738+0000 7f5a57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a33f62a9-e415-4be0-a553-c3c6085e1d3d", "id": 6}] 2024-03-29T10:38:57.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.738+0000 7f5a57fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:57.722 INFO:teuthology.orchestra.run.smithi173.stdout:6 2024-03-29T10:38:57.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 10 librados: watch_flush enter 2024-03-29T10:38:57.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 10 librados: watch_flush exit 2024-03-29T10:38:57.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:38:57.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:38:57.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 10 monclient: shutdown 2024-03-29T10:38:57.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.724 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 >> v1:172.21.15.173:6789/0 conn(0x7f5a70142620 legacy=0x7f5a701e1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 shutdown_connections 2024-03-29T10:38:57.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 >> 172.21.15.173:0/2733771455 conn(0x7f5a70066b00 msgr2=0x7f5a7013de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 shutdown_connections 2024-03-29T10:38:57.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 1 -- 172.21.15.173:0/2733771455 wait complete. 2024-03-29T10:38:57.725 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.739+0000 7f5a75d4b640 1 librados: shutdown 2024-03-29T10:38:57.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 4f6bab91-7ecd-4820-827f-9151d44958dd 7 2024-03-29T10:38:57.736 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.751+0000 7fdf2ea20040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:38:57.804 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.820+0000 7f6834a13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:38:57.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.850+0000 7fe3160fb640 10 monclient: get_monmap_and_config 2024-03-29T10:38:57.834 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.850+0000 7fe3160fb640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.850+0000 7fe3160fb640 10 monclient: monmap: 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:57.851192+0000 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:57.851192+0000 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.850+0000 7fe3160fb640 1 Processor -- start 2024-03-29T10:38:57.835 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.850+0000 7fe3160fb640 1 -- start start 2024-03-29T10:38:57.836 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.850+0000 7fe3160fb640 10 monclient: init 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe31009ac40 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe31009c870 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe3100b4360 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31009dec0 con 0x7fe31009ac40 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31009da30 con 0x7fe31009c870 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3100b39c0 con 0x7fe3100b4360 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe3160fb640 10 monclient(hunting): authenticate will time out at 785.101428s 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30effd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe31009c870 0x7fe31013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34374/0 (socket says 172.21.15.173:34374) 2024-03-29T10:38:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30effd640 1 -- 172.21.15.173:0/616196998 learned_addr learned my addr 172.21.15.173:0/616196998 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717162915 0 0) 0x7fe31009da30 con 0x7fe31009c870 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30e7fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.851+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f8003520 con 0x7fe31009c870 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404305413 0 0) 0x7fe300001110 con 0x7fe31009ac40 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe31009da30 con 0x7fe31009ac40 2024-03-29T10:38:57.846 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708795127 0 0) 0x7fe3100b39c0 con 0x7fe3100b4360 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe300001110 con 0x7fe3100b4360 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 955155883 0 0) 0x7fe2f8003520 con 0x7fe31009c870 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 >> v1:172.21.15.94:6790/0 conn(0x7fe3100b4360 legacy=0x7fe31014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 >> v1:172.21.15.94:6789/0 conn(0x7fe31009ac40 legacy=0x7fe3100b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.847 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100b5160 con 0x7fe31009c870 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe304001690 con 0x7fe31009c870 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: dump: 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.848 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe304001870 con 0x7fe31009c870 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 4 set_mon_vals no callback set 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 1 -- 172.21.15.173:0/616196998 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe304001c80 con 0x7fe31009c870 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe30e7fc640 10 monclient: dump: 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.849 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 10 monclient: shutdown 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 1 -- 172.21.15.173:0/616196998 >> v1:172.21.15.173:6789/0 conn(0x7fe31009c870 legacy=0x7fe31013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.850 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 1 -- 172.21.15.173:0/616196998 shutdown_connections 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 1 -- 172.21.15.173:0/616196998 >> 172.21.15.173:0/616196998 conn(0x7fe310066b00 msgr2=0x7fe310066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 1 -- 172.21.15.173:0/616196998 shutdown_connections 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.852+0000 7fe3160fb640 1 -- 172.21.15.173:0/616196998 wait complete. 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.853+0000 7fe3160fb640 10 monclient: build_initial_monmap 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.853+0000 7fe3160fb640 10 monclient: monmap: 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:57.851 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.853+0000 7fe3160fb640 1 librados: starting msgr at 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.853+0000 7fe3160fb640 1 librados: starting objecter 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.853+0000 7fe3160fb640 1 Processor -- start 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 1 -- start start 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 1 librados: setting wanted keys 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 1 librados: calling monclient init 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient: init 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe31009ac40 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe3100b6090 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe3100b0f40 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3100b5c00 con 0x7fe31009ac40 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3100b73b0 con 0x7fe3100b0f40 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31013a7f0 con 0x7fe3100b6090 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe3160fb640 10 monclient(hunting): authenticate will time out at 785.104473s 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe30ffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe3100b0f40 0x7fe31013a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:34378/0 (socket says 172.21.15.173:34378) 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe30ffff640 1 -- 172.21.15.173:0/3073855771 learned_addr learned my addr 172.21.15.173:0/3073855771 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 client.?.objecter ms_handle_connect 0x7fe3100b0f40 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 client.?.objecter ms_handle_connect 0x7fe31009ac40 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 client.?.objecter ms_handle_connect 0x7fe3100b6090 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162863466 0 0) 0x7fe3100b73b0 con 0x7fe3100b0f40 2024-03-29T10:38:57.853 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.854+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc004eb0 con 0x7fe3100b0f40 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891663468 0 0) 0x7fe3100b5c00 con 0x7fe31009ac40 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3100b73b0 con 0x7fe31009ac40 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74307353 0 0) 0x7fe31013a7f0 con 0x7fe3100b6090 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:57.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): my global_id is 4160 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3100b5c00 con 0x7fe3100b6090 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1570849111 0 0) 0x7fe2fc004eb0 con 0x7fe3100b0f40 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe31013a7f0 con 0x7fe3100b0f40 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe308001c70 con 0x7fe3100b0f40 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): dump: 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.855 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894293467 0 0) 0x7fe3100b73b0 con 0x7fe31009ac40 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2fc004eb0 con 0x7fe31009ac40 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe300001920 con 0x7fe31009ac40 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): dump: 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.856 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 323677330 0 0) 0x7fe3100b5c00 con 0x7fe3100b6090 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3100b73b0 con 0x7fe3100b6090 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe3040019e0 con 0x7fe3100b6090 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient(hunting): dump: 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4084532918 0 0) 0x7fe31013a7f0 con 0x7fe3100b0f40 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 >> v1:172.21.15.94:6790/0 conn(0x7fe3100b6090 legacy=0x7fe3101399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.858 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 >> v1:172.21.15.94:6789/0 conn(0x7fe31009ac40 legacy=0x7fe3100b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 monclient: found mon.b 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100b7620 con 0x7fe3100b0f40 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 10 monclient: _renew_subs 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3100b5840 con 0x7fe3100b0f40 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 10 monclient: _renew_subs 2024-03-29T10:38:57.859 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101428e0 con 0x7fe3100b0f40 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe3160fb640 1 librados: init done 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe308003990 con 0x7fe3100b0f40 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe308003da0 con 0x7fe3100b0f40 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 monclient: dump: 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2024-03-29T10:38:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe30dffb640 4 set_mon_vals no callback set 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe308004350 con 0x7fe3100b0f40 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 4062836338 0 0) 0x7fe308006c80 con 0x7fe3100b0f40 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 10 client.4153.objecter ms_dispatch 0x7fe310000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.855+0000 7fe2f7fff640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.856+0000 7fe3160fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.856+0000 7fe3160fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.856+0000 7fe3160fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.856+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f0005080 con 0x7fe3100b0f40 2024-03-29T10:38:57.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.858+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe3080047e0 con 0x7fe3100b0f40 2024-03-29T10:38:57.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.858+0000 7fe2f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:57.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.858+0000 7fe2f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:57.877 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:57.893+0000 7fdf2ea20040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:38:57.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.930+0000 7fe3160fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4f6bab91-7ecd-4820-827f-9151d44958dd", "id": 7}] 2024-03-29T10:38:57.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.930+0000 7fe3160fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4f6bab91-7ecd-4820-827f-9151d44958dd", "id": 7}] 2024-03-29T10:38:57.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.930+0000 7fe3160fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:38:57.915 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.930+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4f6bab91-7ecd-4820-827f-9151d44958dd", "id": 7} v 0) v1 -- 0x7fe2f00058b0 con 0x7fe3100b0f40 2024-03-29T10:38:57.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.934+0000 7fe2f7fff640 1 -- 172.21.15.173:0/3073855771 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f6bab91-7ecd-4820-827f-9151d44958dd", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1601340879 0 1371665406) 0x7fe308006620 con 0x7fe3100b0f40 2024-03-29T10:38:57.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.934+0000 7fe2f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4f6bab91-7ecd-4820-827f-9151d44958dd", "id": 7}] 2024-03-29T10:38:57.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.934+0000 7fe2f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:57.918 INFO:teuthology.orchestra.run.smithi173.stdout:7 2024-03-29T10:38:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 10 librados: watch_flush enter 2024-03-29T10:38:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 10 librados: watch_flush exit 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 10 monclient: shutdown 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 >> v1:172.21.15.173:6789/0 conn(0x7fe3100b0f40 legacy=0x7fe31013a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 shutdown_connections 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 >> 172.21.15.173:0/3073855771 conn(0x7fe310066b00 msgr2=0x7fe31013c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 shutdown_connections 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 1 -- 172.21.15.173:0/3073855771 wait complete. 2024-03-29T10:38:57.920 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-29T10:38:57.935+0000 7fe3160fb640 1 librados: shutdown 2024-03-29T10:38:57.929 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:38:57.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:38:57.932 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:38:57.932 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:38:57.932 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:38:57.932 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:57.947+0000 7f6834a13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:38:57.934 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:38:57.934 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:38:57.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:38:57.937 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:38:57.937 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:38:57.937 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:38:57.939 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:38:57.939 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:38:57.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:38:57.942 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:38:57.943 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:38:57.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:38:57.945 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:38:57.946 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:38:57.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:38:57.948 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:38:57.948 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:38:57.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:38:57.951 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:38:57.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:57.994 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:58.011+0000 7f6834a13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:38:57.996 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:58.013+0000 7fdf2ea20040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:38:58.057 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:58.073+0000 7fdf2ea20040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:38:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.094+0000 7f2c57d3d640 10 monclient: get_monmap_and_config 2024-03-29T10:38:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.094+0000 7f2c57d3d640 10 monclient: build_initial_monmap 2024-03-29T10:38:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.094+0000 7f2c57d3d640 10 monclient: monmap: 2024-03-29T10:38:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T10:38:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:58.078 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:58.095322+0000 2024-03-29T10:38:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:58.095322+0000 2024-03-29T10:38:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:58.079 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:58.079 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 1 Processor -- start 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 1 -- start start 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient: init 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f2c500b8240 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c500b1f60 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c5009c8c0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:58.087 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5009bd80 con 0x7f2c500b1f60 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5009b960 con 0x7f2c5009c8c0 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c500b2ec0 con 0x7f2c500b8240 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c57d3d640 10 monclient(hunting): authenticate will time out at 789.363930s 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c552d8640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2c500b1f60 0x7f2c5009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51124/0 (socket says 172.21.15.94:51124) 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.095+0000 7f2c552d8640 1 -- 172.21.15.94:0/2500603391 learned_addr learned my addr 172.21.15.94:0/2500603391 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613495045 0 0) 0x7f2c5009bd80 con 0x7f2c500b1f60 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _negotiate 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): my global_id is 4185 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c3c003520 con 0x7f2c500b1f60 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333611321 0 0) 0x7f2c500b2ec0 con 0x7f2c500b8240 2024-03-29T10:38:58.088 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _negotiate 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c5009bd80 con 0x7f2c500b8240 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770893059 0 0) 0x7f2c5009b960 con 0x7f2c5009c8c0 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _negotiate 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient(hunting): my global_id is 4180 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c500b2ec0 con 0x7f2c5009c8c0 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1888723194 0 0) 0x7f2c5009bd80 con 0x7f2c500b8240 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 >> v1:172.21.15.173:6789/0 conn(0x7f2c5009c8c0 legacy=0x7f2c5014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 >> v1:172.21.15.94:6789/0 conn(0x7f2c500b1f60 legacy=0x7f2c5009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:58.089 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: _finish_hunting 0 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 monclient: found mon.noname-b 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5013fb60 con 0x7f2c500b8240 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: _finish_auth 0 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: _check_auth_tickets 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2c40001670 con 0x7f2c500b8240 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: dump: 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:58.090 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c57d3d640 5 monclient: authenticate success, global_id 4187 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c57d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c40001850 con 0x7f2c500b8240 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:58.091 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 1 -- 172.21.15.94:0/2500603391 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2c40001ca0 con 0x7f2c500b8240 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c57d3d640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c57d3d640 4 set_mon_vals no callback set 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.096+0000 7f2c54ad7640 10 monclient: dump: 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:58.092 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 10 monclient: shutdown 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 1 -- 172.21.15.94:0/2500603391 >> v1:172.21.15.94:6790/0 conn(0x7f2c500b8240 legacy=0x7f2c5013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 1 -- 172.21.15.94:0/2500603391 shutdown_connections 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 1 -- 172.21.15.94:0/2500603391 >> 172.21.15.94:0/2500603391 conn(0x7f2c50066b00 msgr2=0x7f2c50066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 1 -- 172.21.15.94:0/2500603391 shutdown_connections 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 1 -- 172.21.15.94:0/2500603391 wait complete. 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 10 monclient: build_initial_monmap 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.097+0000 7f2c57d3d640 10 monclient: monmap: 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 0 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 0.000000 2024-03-29T10:38:58.093 INFO:teuthology.orchestra.run.smithi094.stderr:created 0.000000 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 1 librados: starting msgr at 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 1 librados: starting objecter 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 1 Processor -- start 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 1 -- start start 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 1 librados: setting wanted keys 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 1 librados: calling monclient init 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 10 monclient: init 2024-03-29T10:38:58.094 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f2c5009c8c0 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f2c500b8240 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f2c50132750 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c57d3d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c50133aa0 con 0x7f2c500b8240 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c552d8640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2c500b8240 0x7f2c50134760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51138/0 (socket says 172.21.15.94:51138) 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c55ad9640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f2c5009c8c0 0x7f2c50144c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:37908/0 (socket says 172.21.15.94:37908) 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.098+0000 7f2c552d8640 1 -- 172.21.15.94:0/1344609971 learned_addr learned my addr 172.21.15.94:0/1344609971 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c50132b50 con 0x7f2c50132750 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c50133d10 con 0x7f2c5009c8c0 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 10 monclient(hunting): authenticate will time out at 789.367150s 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 client.?.objecter ms_handle_connect 0x7f2c5009c8c0 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:58.095 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 client.?.objecter ms_handle_connect 0x7f2c500b8240 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 client.?.objecter ms_handle_connect 0x7f2c50132750 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052098107 0 0) 0x7f2c50133d10 con 0x7f2c5009c8c0 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c34004eb0 con 0x7f2c5009c8c0 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402075195 0 0) 0x7f2c50133aa0 con 0x7f2c500b8240 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:58.096 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c50133d10 con 0x7f2c500b8240 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190677432 0 0) 0x7f2c50132b50 con 0x7f2c50132750 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c50133aa0 con 0x7f2c50132750 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224679845 0 0) 0x7f2c50133d10 con 0x7f2c500b8240 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c50132b50 con 0x7f2c500b8240 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2c4c0018d0 con 0x7f2c500b8240 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): dump: 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:58.097 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2940739465 0 0) 0x7f2c34004eb0 con 0x7f2c5009c8c0 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c50133d10 con 0x7f2c5009c8c0 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807390235 0 0) 0x7f2c50133aa0 con 0x7f2c50132750 2024-03-29T10:38:58.098 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c34004eb0 con 0x7f2c50132750 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2c40001be0 con 0x7f2c5009c8c0 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): dump: 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:58.099 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2c48001b50 con 0x7f2c50132750 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient(hunting): dump: 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.100 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 622512972 0 0) 0x7f2c50132b50 con 0x7f2c500b8240 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 >> v1:172.21.15.94:6790/0 conn(0x7f2c5009c8c0 legacy=0x7f2c50144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 >> v1:172.21.15.173:6789/0 conn(0x7f2c50132750 legacy=0x7f2c50134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 monclient: found mon.a 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c50133ff0 con 0x7f2c500b8240 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 10 monclient: _renew_subs 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:58.101 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c501342d0 con 0x7f2c500b8240 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 10 monclient: _renew_subs 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c50131c70 con 0x7f2c500b8240 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c57d3d640 1 librados: init done 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c4c0047e0 con 0x7f2c500b8240 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2c4c004af0 con 0x7f2c500b8240 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c47fff640 4 set_mon_vals no callback set 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 10 monclient: dump: 2024-03-29T10:38:58.102 INFO:teuthology.orchestra.run.smithi094.stderr:epoch 1 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.099+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2c4c005020 con 0x7f2c500b8240 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.100+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f2c4c0059a0 con 0x7f2c500b8240 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.100+0000 7f2c467fc640 10 client.4188.objecter ms_dispatch 0x7f2c50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.100+0000 7f2c467fc640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.100+0000 7f2c467fc640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:58.103 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.100+0000 7f2c467fc640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T10:38:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.101+0000 7f2c57d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.101+0000 7f2c57d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.101+0000 7f2c57d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.101+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c38005080 con 0x7f2c500b8240 2024-03-29T10:38:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.103+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2c4c005be0 con 0x7f2c500b8240 2024-03-29T10:38:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.103+0000 7f2c467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:58.104 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.103+0000 7f2c467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:58.127 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:58.144+0000 7f6834a13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:38:58.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.163+0000 7f2c57d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:58.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.163+0000 7f2c57d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:58.147 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.163+0000 7f2c57d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:38:58.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.163+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c38005330 con 0x7f2c500b8240 2024-03-29T10:38:58.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.163+0000 7f2c467fc640 1 -- 172.21.15.94:0/1344609971 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f2c4c005380 con 0x7f2c500b8240 2024-03-29T10:38:58.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.163+0000 7f2c467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:58.148 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.163+0000 7f2c467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:58.148 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-03-29T10:38:58.148 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:58.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 10 librados: watch_flush enter 2024-03-29T10:38:58.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 10 librados: watch_flush exit 2024-03-29T10:38:58.149 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 10 client.4188.objecter successfully canceled tick 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 10 monclient: shutdown 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 >> v1:172.21.15.94:6789/0 conn(0x7f2c500b8240 legacy=0x7f2c50134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 shutdown_connections 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 >> 172.21.15.94:0/1344609971 conn(0x7f2c50066b00 msgr2=0x7f2c5013dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 shutdown_connections 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.164+0000 7f2c57d3d640 1 -- 172.21.15.94:0/1344609971 wait complete. 2024-03-29T10:38:58.150 INFO:teuthology.orchestra.run.smithi094.stderr:2024-03-29T10:38:58.165+0000 7f2c57d3d640 1 librados: shutdown 2024-03-29T10:38:58.159 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:38:58.159 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:38:58.170 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:58.186+0000 7fdf2ea20040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:38:58.225 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:58.241+0000 7f6834a13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:38:58.265 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:58.280+0000 7fdf2ea20040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:38:58.379 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:58.396+0000 7f6834a13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:38:58.420 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:58.435+0000 7fdf2ea20040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:38:58.421 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:58.437+0000 7f6834a13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:38:58.461 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:58.477+0000 7fdf2ea20040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:38:58.533 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-03-29T10:38:58.549+0000 7f6d19e24640 -1 Falling back to public interface 2024-03-29T10:38:58.547 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-29T10:38:58.563+0000 7f1bae2d5640 -1 Falling back to public interface 2024-03-29T10:38:58.549 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-29T10:38:58.565+0000 7f462901c640 -1 Falling back to public interface 2024-03-29T10:38:58.550 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-03-29T10:38:58.566+0000 7f218ccd9640 -1 Falling back to public interface 2024-03-29T10:38:58.559 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-03-29T10:38:58.575+0000 7f0c57489640 -1 Falling back to public interface 2024-03-29T10:38:58.563 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-29T10:38:58.579+0000 7f1610919640 -1 Falling back to public interface 2024-03-29T10:38:58.564 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-03-29T10:38:58.580+0000 7fc1aa40f640 -1 Falling back to public interface 2024-03-29T10:38:58.565 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-29T10:38:58.581+0000 7fb907817640 -1 Falling back to public interface 2024-03-29T10:38:58.574 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:58.590+0000 7f6834a13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:38:58.617 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:58.633+0000 7fdf2ea20040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:38:58.965 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:59.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.071+0000 7f2bab10e640 10 monclient: get_monmap_and_config 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.071+0000 7f2bab10e640 10 monclient: build_initial_monmap 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.071+0000 7f2bab10e640 10 monclient: monmap: 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:59.072693+0000 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:59.072693+0000 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:38:59.056 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.072+0000 7f2bab10e640 1 Processor -- start 2024-03-29T10:38:59.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.072+0000 7f2bab10e640 1 -- start start 2024-03-29T10:38:59.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.072+0000 7f2bab10e640 10 monclient: init 2024-03-29T10:38:59.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:59.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba41422b0 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba4142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba40b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba4142c80 con 0x7f2ba4142690 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40aae40 con 0x7f2ba41422b0 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba413fbd0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient(hunting): authenticate will time out at 790.341372s 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2ba4142690 0x7f2ba40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51152/0 (socket says 172.21.15.94:51152) 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9bfff640 1 -- 172.21.15.94:0/1410793673 learned_addr learned my addr 172.21.15.94:0/1410793673 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631081799 0 0) 0x7f2ba413fbd0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:59.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b8c003520 con 0x7f2ba40b17d0 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038993514 0 0) 0x7f2ba4142c80 con 0x7f2ba4142690 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba413fbd0 con 0x7f2ba4142690 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562369987 0 0) 0x7f2b8c003520 con 0x7f2ba40b17d0 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 >> v1:172.21.15.173:6789/0 conn(0x7f2ba41422b0 legacy=0x7f2ba413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 >> v1:172.21.15.94:6789/0 conn(0x7f2ba4142690 legacy=0x7f2ba40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:38:59.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba4151940 con 0x7f2ba40b17d0 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2b9c001a10 con 0x7f2ba40b17d0 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: dump: 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.068 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 5 monclient: authenticate success, global_id 4193 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b9c001b70 con 0x7f2ba40b17d0 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 1 -- 172.21.15.94:0/1410793673 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2b9c001f40 con 0x7f2ba40b17d0 2024-03-29T10:38:59.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2bab10e640 4 set_mon_vals no callback set 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.073+0000 7f2b9b7fe640 10 monclient: dump: 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:59.070 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.074+0000 7f2bab10e640 10 monclient: shutdown 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.074+0000 7f2bab10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.074+0000 7f2bab10e640 1 -- 172.21.15.94:0/1410793673 >> v1:172.21.15.94:6790/0 conn(0x7f2ba40b17d0 legacy=0x7f2ba40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.074+0000 7f2bab10e640 1 -- 172.21.15.94:0/1410793673 shutdown_connections 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.074+0000 7f2bab10e640 1 -- 172.21.15.94:0/1410793673 >> 172.21.15.94:0/1410793673 conn(0x7f2ba4066a00 msgr2=0x7f2ba4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.074+0000 7f2bab10e640 1 -- 172.21.15.94:0/1410793673 shutdown_connections 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.074+0000 7f2bab10e640 1 -- 172.21.15.94:0/1410793673 wait complete. 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 10 monclient: build_initial_monmap 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 10 monclient: monmap: 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:38:59.071 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 1 librados: starting msgr at 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 1 librados: starting objecter 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 1 Processor -- start 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 1 -- start start 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 1 librados: setting wanted keys 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 1 librados: calling monclient init 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.075+0000 7f2bab10e640 10 monclient: init 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:59.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 10 monclient(hunting): picked mon.noname-c con 0x7f2ba40b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 10 monclient(hunting): picked mon.noname-a con 0x7f2ba41422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 10 monclient(hunting): picked mon.noname-b con 0x7f2ba4142690 addr v1:172.21.15.173:6789/0 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba4131bb0 con 0x7f2ba41422b0 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba414e420 con 0x7f2ba4142690 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40abba0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2bab10e640 10 monclient(hunting): authenticate will time out at 790.344291s 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2ba8eaa640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f2ba40b17d0 0x7f2ba412f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:37920/0 (socket says 172.21.15.94:37920) 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b9bfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f2ba41422b0 0x7f2ba414d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:51162/0 (socket says 172.21.15.94:51162) 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2ba8eaa640 1 -- 172.21.15.94:0/32234087 learned_addr learned my addr 172.21.15.94:0/32234087 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 client.?.objecter ms_handle_connect 0x7f2ba40b17d0 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 client.?.objecter ms_handle_connect 0x7f2ba41422b0 2024-03-29T10:38:59.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 client.?.objecter ms_handle_connect 0x7f2ba4142690 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980541148 0 0) 0x7f2ba40abba0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): my global_id is 4196 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b90004eb0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007254614 0 0) 0x7f2ba4131bb0 con 0x7f2ba41422b0 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:59.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba40abba0 con 0x7f2ba41422b0 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319990678 0 0) 0x7f2ba414e420 con 0x7f2ba4142690 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _negotiate 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba4131bb0 con 0x7f2ba4142690 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224807949 0 0) 0x7f2b90004eb0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba414e420 con 0x7f2ba40b17d0 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2ba0001a10 con 0x7f2ba40b17d0 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): dump: 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:38:59.075 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839544602 0 0) 0x7f2ba40abba0 con 0x7f2ba41422b0 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b90004eb0 con 0x7f2ba41422b0 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2b94001810 con 0x7f2ba41422b0 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:59.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): dump: 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1765272688 0 0) 0x7f2ba4131bb0 con 0x7f2ba4142690 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba40abba0 con 0x7f2ba4142690 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2b9c001730 con 0x7f2ba4142690 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:59.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 10 monclient(hunting): dump: 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116466664 0 0) 0x7f2ba414e420 con 0x7f2ba40b17d0 2024-03-29T10:38:59.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 >> v1:172.21.15.173:6789/0 conn(0x7f2ba4142690 legacy=0x7f2ba414dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.076+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 >> v1:172.21.15.94:6789/0 conn(0x7f2ba41422b0 legacy=0x7f2ba414d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: _finish_hunting 0 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 1 monclient: found mon.c 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40abe80 con 0x7f2ba40b17d0 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: _finish_auth 0 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: _check_auth_tickets 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 5 monclient: authenticate success, global_id 4196 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 monclient: _renew_subs 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba4130080 con 0x7f2ba40b17d0 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:59.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 monclient: _renew_subs 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba4132220 con 0x7f2ba40b17d0 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 1 librados: init done 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ba0001c50 con 0x7f2ba40b17d0 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2ba0003a60 con 0x7f2ba40b17d0 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b9affd640 4 set_mon_vals no callback set 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 monclient: dump: 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:38:59.080 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2ba0003fd0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f2ba0006a90 con 0x7f2ba40b17d0 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 10 client.4196.objecter ms_dispatch 0x7f2ba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2b997fa640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:59.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:59.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.077+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b84005080 con 0x7f2ba40b17d0 2024-03-29T10:38:59.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.080+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2ba00063c0 con 0x7f2ba40b17d0 2024-03-29T10:38:59.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.080+0000 7f2b997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:59.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.080+0000 7f2b997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:59.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.141+0000 7f2bab10e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:59.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.141+0000 7f2bab10e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:59.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.141+0000 7f2bab10e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:38:59.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.141+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b84005330 con 0x7f2ba40b17d0 2024-03-29T10:38:59.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2b997fa640 1 -- 172.21.15.94:0/32234087 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f2ba0006560 con 0x7f2ba40b17d0 2024-03-29T10:38:59.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2b997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:59.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2b997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:59.126 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:38:59.126 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2bab10e640 10 librados: watch_flush enter 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2bab10e640 10 librados: watch_flush exit 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2bab10e640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2bab10e640 10 client.4196.objecter successfully canceled tick 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2bab10e640 10 monclient: shutdown 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2bab10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.142+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 >> v1:172.21.15.94:6790/0 conn(0x7f2ba40b17d0 legacy=0x7f2ba412f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.143+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 shutdown_connections 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.143+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 >> 172.21.15.94:0/32234087 conn(0x7f2ba4066a00 msgr2=0x7f2ba413b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.143+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 shutdown_connections 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.143+0000 7f2bab10e640 1 -- 172.21.15.94:0/32234087 wait complete. 2024-03-29T10:38:59.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:38:59.143+0000 7f2bab10e640 1 librados: shutdown 2024-03-29T10:38:59.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:59.380 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:59.396+0000 7f6834a13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:38:59.431 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:59.447+0000 7fdf2ea20040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:38:59.650 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:59.666+0000 7f6834a13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:38:59.704 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:59.720+0000 7f6834a13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:38:59.704 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:59.720+0000 7fdf2ea20040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:38:59.755 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:59.771+0000 7fdf2ea20040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:38:59.817 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:38:59.833+0000 7f6834a13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:38:59.871 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-29T10:38:59.887+0000 7fdf2ea20040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:39:00.830 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:39:00.846+0000 7f67d535a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:39:00.830 INFO:tasks.ceph.mgr.y.smithi094.stderr:2024-03-29T10:39:00.846+0000 7f67d535a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:39:01.828 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-29T10:39:01.844+0000 7f1610919640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:39:02.018 INFO:tasks.ceph.osd.0.smithi094.stderr:2024-03-29T10:39:02.034+0000 7f218ccd9640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:39:02.049 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-29T10:39:02.065+0000 7f462901c640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:39:02.077 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-29T10:39:02.093+0000 7fb907817640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:39:02.078 INFO:tasks.ceph.osd.1.smithi094.stderr:2024-03-29T10:39:02.094+0000 7f0c57489640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:39:02.079 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-29T10:39:02.096+0000 7f1bae2d5640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:39:02.084 INFO:tasks.ceph.osd.2.smithi094.stderr:2024-03-29T10:39:02.100+0000 7f6d19e24640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:39:02.085 INFO:tasks.ceph.osd.3.smithi094.stderr:2024-03-29T10:39:02.101+0000 7fc1aa40f640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:39:05.943 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:06.092 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.108+0000 7f45d7831640 10 monclient: get_monmap_and_config 2024-03-29T10:39:06.092 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.108+0000 7f45d7831640 10 monclient: build_initial_monmap 2024-03-29T10:39:06.092 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.108+0000 7f45d7831640 10 monclient: monmap: 2024-03-29T10:39:06.092 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:06.109399+0000 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:06.109399+0000 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:06.093 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:06.094 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.109+0000 7f45d7831640 1 Processor -- start 2024-03-29T10:39:06.094 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.109+0000 7f45d7831640 1 -- start start 2024-03-29T10:39:06.094 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.109+0000 7f45d7831640 10 monclient: init 2024-03-29T10:39:06.094 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.110+0000 7f45d7831640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:06.094 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.110+0000 7f45d7831640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:06.094 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.110+0000 7f45d7831640 10 monclient(hunting): picked mon.noname-c con 0x7f45d01457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:06.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.110+0000 7f45d7831640 10 monclient(hunting): picked mon.noname-b con 0x7f45d013f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:06.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.110+0000 7f45d7831640 10 monclient(hunting): picked mon.noname-a con 0x7f45d00a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d7831640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00aaef0 con 0x7f45d00a9d70 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d5dce640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f45d00a9d70 0x7f45d014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:47924/0 (socket says 172.21.15.94:47924) 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d5dce640 1 -- 172.21.15.94:0/3857620103 learned_addr learned my addr 172.21.15.94:0/3857620103 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d7831640 1 -- 172.21.15.94:0/3857620103 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d0145bd0 con 0x7f45d01457f0 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d7831640 1 -- 172.21.15.94:0/3857620103 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d0145eb0 con 0x7f45d013f650 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d7831640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d7831640 10 monclient(hunting): authenticate will time out at 797.379238s 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45d55cd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f45d01457f0 0x7f45d013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:39132/0 (socket says 172.21.15.94:39132) 2024-03-29T10:39:06.097 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782602513 0 0) 0x7f45d0145bd0 con 0x7f45d01457f0 2024-03-29T10:39:06.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:06.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:06.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:06.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45c7fff640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:39:06.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.111+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45bc003520 con 0x7f45d01457f0 2024-03-29T10:39:06.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419069609 0 0) 0x7f45d00aaef0 con 0x7f45d00a9d70 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d0145bd0 con 0x7f45d00a9d70 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3123368222 0 0) 0x7f45bc003520 con 0x7f45d01457f0 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 >> v1:172.21.15.94:6790/0 conn(0x7f45d013f650 legacy=0x7f45d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 >> v1:172.21.15.94:6789/0 conn(0x7f45d00a9d70 legacy=0x7f45d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:06.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 monclient: found mon.noname-c 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d0146340 con 0x7f45d01457f0 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f45cc001690 con 0x7f45d01457f0 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: dump: 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:06.100 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:06.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45d7831640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45d7831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45cc0017f0 con 0x7f45d01457f0 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45d7831640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45d7831640 4 set_mon_vals no callback set 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 1 -- 172.21.15.94:0/3857620103 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f45cc001c40 con 0x7f45d01457f0 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.112+0000 7f45c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:06.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45c7fff640 10 monclient: dump: 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:06.103 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:06.104 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:06.113 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45d7831640 10 monclient: shutdown 2024-03-29T10:39:06.113 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45d7831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45d7831640 1 -- 172.21.15.94:0/3857620103 >> v1:172.21.15.173:6789/0 conn(0x7f45d01457f0 legacy=0x7f45d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45d7831640 1 -- 172.21.15.94:0/3857620103 shutdown_connections 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45d7831640 1 -- 172.21.15.94:0/3857620103 >> 172.21.15.94:0/3857620103 conn(0x7f45d0066a00 msgr2=0x7f45d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45d7831640 1 -- 172.21.15.94:0/3857620103 shutdown_connections 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.113+0000 7f45d7831640 1 -- 172.21.15.94:0/3857620103 wait complete. 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.114+0000 7f45d7831640 10 monclient: build_initial_monmap 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 10 monclient: monmap: 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:06.114 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 1 librados: starting msgr at 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 1 librados: starting objecter 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 1 Processor -- start 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 1 -- start start 2024-03-29T10:39:06.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 1 librados: setting wanted keys 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 1 librados: calling monclient init 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.115+0000 7f45d7831640 10 monclient: init 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.116+0000 7f45d7831640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.116+0000 7f45d7831640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.116+0000 7f45d7831640 10 monclient(hunting): picked mon.noname-c con 0x7f45d00a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.116+0000 7f45d7831640 10 monclient(hunting): picked mon.noname-b con 0x7f45d013f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.116+0000 7f45d7831640 10 monclient(hunting): picked mon.noname-a con 0x7f45d01457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.116+0000 7f45d4dcc640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f45d013f650 0x7f45d00ae260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:39266/0 (socket says 172.21.15.94:39266) 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45d4dcc640 1 -- 172.21.15.94:0/1216627444 learned_addr learned my addr 172.21.15.94:0/1216627444 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00afbf0 con 0x7f45d01457f0 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00afe60 con 0x7f45d013f650 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d01d7320 con 0x7f45d00a9d70 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45d7831640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:06.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45d7831640 10 monclient(hunting): authenticate will time out at 797.385601s 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 10 client.?.objecter ms_handle_connect 0x7f45d013f650 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 10 client.?.objecter ms_handle_connect 0x7f45d01457f0 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 10 client.?.objecter ms_handle_connect 0x7f45d00a9d70 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921820619 0 0) 0x7f45d00afe60 con 0x7f45d013f650 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.117+0000 7f45c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b4004eb0 con 0x7f45d013f650 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849934187 0 0) 0x7f45d00afbf0 con 0x7f45d01457f0 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:06.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d00afe60 con 0x7f45d01457f0 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391784470 0 0) 0x7f45d01d7320 con 0x7f45d00a9d70 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d00afbf0 con 0x7f45d00a9d70 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4104016671 0 0) 0x7f45b4004eb0 con 0x7f45d013f650 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.118+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45d01d7320 con 0x7f45d013f650 2024-03-29T10:39:06.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f45c0001910 con 0x7f45d013f650 2024-03-29T10:39:06.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:06.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:06.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:06.119 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:06.119 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:06.119 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.119 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.120 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:06.120 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:06.120 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:06.120 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:06.120 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:06.120 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:06.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4102373415 0 0) 0x7f45d01d7320 con 0x7f45d013f650 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 >> v1:172.21.15.94:6790/0 conn(0x7f45d00a9d70 legacy=0x7f45d01d7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 >> v1:172.21.15.94:6789/0 conn(0x7f45d01457f0 legacy=0x7f45d01e2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 1 monclient: found mon.b 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00b12f0 con 0x7f45d013f650 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:06.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.119+0000 7f45c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 5 monclient: authenticate success, global_id 4237 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 10 monclient: _renew_subs 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d00adc90 con 0x7f45d013f650 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 10 monclient: _renew_subs 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c0001a70 con 0x7f45d013f650 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d01d8130 con 0x7f45d013f650 2024-03-29T10:39:06.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45d7831640 1 librados: init done 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f45c00038b0 con 0x7f45d013f650 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45c5ffb640 10 monclient: dump: 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:06.123 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.120+0000 7f45c77fe640 4 set_mon_vals no callback set 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.122+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f45c0003a10 con 0x7f45d013f650 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45d55cd640 10 monclient: get_auth_request con 0x7f45b406df00 auth_method 0 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45d7831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45d7831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45d7831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d01d74b0 con 0x7f45d013f650 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f45c0084a20 con 0x7f45d013f650 2024-03-29T10:39:06.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45c5ffb640 10 client.4237.objecter ms_dispatch 0x7f45d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:06.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45c5ffb640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:06.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45c5ffb640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:06.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45c5ffb640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T10:39:06.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.123+0000 7f45c5ffb640 10 client.4237.objecter ms_handle_connect 0x7f45b406df00 2024-03-29T10:39:06.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.130+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45c0050360 con 0x7f45d013f650 2024-03-29T10:39:06.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.130+0000 7f45c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:06.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.130+0000 7f45c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:06.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.549+0000 7f45d7831640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:06.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.549+0000 7f45d7831640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:06.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.549+0000 7f45d7831640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:39:06.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.549+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45d0075f10 con 0x7f45d013f650 2024-03-29T10:39:06.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.550+0000 7f45c5ffb640 1 -- 172.21.15.94:0/1216627444 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f45c0052290 con 0x7f45d013f650 2024-03-29T10:39:06.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.550+0000 7f45c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:06.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.550+0000 7f45c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:06.535 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:06.535 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:06.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 10 librados: watch_flush enter 2024-03-29T10:39:06.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 10 librados: watch_flush exit 2024-03-29T10:39:06.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T10:39:06.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 10 client.4237.objecter successfully canceled tick 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 >> v1:172.21.15.94:6816/3829934808 conn(0x7f45b406df00 legacy=0x7f45b40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 10 monclient: shutdown 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.554+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 >> v1:172.21.15.173:6789/0 conn(0x7f45d013f650 legacy=0x7f45d00ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.555+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 shutdown_connections 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.555+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 >> 172.21.15.94:0/1216627444 conn(0x7f45d0066a00 msgr2=0x7f45d00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.555+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 shutdown_connections 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.555+0000 7f45d7831640 1 -- 172.21.15.94:0/1216627444 wait complete. 2024-03-29T10:39:06.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:06.555+0000 7f45d7831640 1 librados: shutdown 2024-03-29T10:39:06.554 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:13.361 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.534+0000 7f29a06fb640 10 monclient: get_monmap_and_config 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.534+0000 7f29a06fb640 10 monclient: build_initial_monmap 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.534+0000 7f29a06fb640 10 monclient: monmap: 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:13.534943+0000 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:13.534943+0000 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:13.518 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:13.519 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:13.519 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:13.519 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:13.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.535+0000 7f29a06fb640 1 Processor -- start 2024-03-29T10:39:13.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.535+0000 7f29a06fb640 1 -- start start 2024-03-29T10:39:13.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.535+0000 7f29a06fb640 10 monclient: init 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 10 monclient(hunting): picked mon.noname-b con 0x7f29981457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 10 monclient(hunting): picked mon.noname-c con 0x7f299813f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 10 monclient(hunting): picked mon.noname-a con 0x7f29980a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29980aaef0 con 0x7f29980a9d70 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2998145bd0 con 0x7f299813f650 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2998145eb0 con 0x7f29981457f0 2024-03-29T10:39:13.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:13.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f29a06fb640 10 monclient(hunting): authenticate will time out at 804.804542s 2024-03-29T10:39:13.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f299e497640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f29981457f0 0x7f299813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:45788/0 (socket says 172.21.15.94:45788) 2024-03-29T10:39:13.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.536+0000 7f299e497640 1 -- 172.21.15.94:0/4112049502 learned_addr learned my addr 172.21.15.94:0/4112049502 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732042045 0 0) 0x7f2998145bd0 con 0x7f299813f650 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _negotiate 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2984003520 con 0x7f299813f650 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841960786 0 0) 0x7f2998145eb0 con 0x7f29981457f0 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _negotiate 2024-03-29T10:39:13.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2998145bd0 con 0x7f29981457f0 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288493885 0 0) 0x7f29980aaef0 con 0x7f29980a9d70 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _negotiate 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2998145eb0 con 0x7f29980a9d70 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589619652 0 0) 0x7f2984003520 con 0x7f299813f650 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 >> v1:172.21.15.94:6790/0 conn(0x7f29981457f0 legacy=0x7f299813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 >> v1:172.21.15.94:6789/0 conn(0x7f29980a9d70 legacy=0x7f299814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient: _finish_hunting 0 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 monclient: found mon.noname-c 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2998146340 con 0x7f299813f650 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient: _finish_auth 0 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient: _check_auth_tickets 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f29940015c0 con 0x7f299813f650 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f299d495640 10 monclient: dump: 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:13.524 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f29a06fb640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.537+0000 7f29a06fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2994001720 con 0x7f299813f650 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f299d495640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f299d495640 1 -- 172.21.15.94:0/4112049502 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2994001b30 con 0x7f299813f650 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f299d495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:13.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f299d495640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f299d495640 10 monclient: dump: 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f29a06fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:13.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f29a06fb640 4 set_mon_vals no callback set 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f29a06fb640 10 monclient: shutdown 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f29a06fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.538+0000 7f29a06fb640 1 -- 172.21.15.94:0/4112049502 >> v1:172.21.15.173:6789/0 conn(0x7f299813f650 legacy=0x7f29980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.539+0000 7f29a06fb640 1 -- 172.21.15.94:0/4112049502 shutdown_connections 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.539+0000 7f29a06fb640 1 -- 172.21.15.94:0/4112049502 >> 172.21.15.94:0/4112049502 conn(0x7f2998066a00 msgr2=0x7f2998066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.539+0000 7f29a06fb640 1 -- 172.21.15.94:0/4112049502 shutdown_connections 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.539+0000 7f29a06fb640 1 -- 172.21.15.94:0/4112049502 wait complete. 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 10 monclient: build_initial_monmap 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 10 monclient: monmap: 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:13.531 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 1 librados: starting msgr at 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 1 librados: starting objecter 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 1 Processor -- start 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 1 -- start start 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 1 librados: setting wanted keys 2024-03-29T10:39:13.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 1 librados: calling monclient init 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.540+0000 7f29a06fb640 10 monclient: init 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 10 monclient(hunting): picked mon.noname-a con 0x7f29980a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 10 monclient(hunting): picked mon.noname-c con 0x7f299813f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 10 monclient(hunting): picked mon.noname-b con 0x7f29981457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2998142210 con 0x7f29980a9d70 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2998142e60 con 0x7f29981457f0 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2998137b10 con 0x7f299813f650 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f29a06fb640 10 monclient(hunting): authenticate will time out at 804.809938s 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f299e497640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f29980a9d70 0x7f2998141940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57536/0 (socket says 172.21.15.94:57536) 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.541+0000 7f299e497640 1 -- 172.21.15.94:0/1612654508 learned_addr learned my addr 172.21.15.94:0/1612654508 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:13.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 client.?.objecter ms_handle_connect 0x7f29980a9d70 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044044640 0 0) 0x7f2998142210 con 0x7f29980a9d70 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f297c004eb0 con 0x7f29980a9d70 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 client.?.objecter ms_handle_connect 0x7f299813f650 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510151447 0 0) 0x7f297c004eb0 con 0x7f29980a9d70 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2998142210 con 0x7f29980a9d70 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2988001910 con 0x7f29980a9d70 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:13.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): dump: 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.542+0000 7f298effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1059921338 0 0) 0x7f2998142210 con 0x7f29980a9d70 2024-03-29T10:39:13.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 >> v1:172.21.15.94:6790/0 conn(0x7f299813f650 legacy=0x7f2998142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 >> v1:172.21.15.173:6789/0 conn(0x7f29981457f0 legacy=0x7f2998071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 1 monclient: found mon.a 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2998144910 con 0x7f29980a9d70 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: _finish_auth 0 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 5 monclient: authenticate success, global_id 4248 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 10 monclient: _renew_subs 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2998144bc0 con 0x7f29980a9d70 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:13.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 10 monclient: _renew_subs 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29981db790 con 0x7f29980a9d70 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f29a06fb640 1 librados: init done 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29880047e0 con 0x7f29980a9d70 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2988004af0 con 0x7f29980a9d70 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f298effd640 10 monclient: dump: 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:13.537 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.543+0000 7f299cc94640 4 set_mon_vals no callback set 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.545+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f298801aac0 con 0x7f29980a9d70 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f2988084b10 con 0x7f29980a9d70 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f298effd640 10 client.4248.objecter ms_dispatch 0x7f2998000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f298effd640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f298effd640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f298effd640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f299dc96640 10 monclient: get_auth_request con 0x7f297c06dd40 auth_method 0 2024-03-29T10:39:13.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f29a06fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:13.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f29a06fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:13.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f29a06fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:13.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2980005080 con 0x7f29980a9d70 2024-03-29T10:39:13.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.546+0000 7f298effd640 10 client.4248.objecter ms_handle_connect 0x7f297c06dd40 2024-03-29T10:39:13.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.555+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2988050220 con 0x7f29980a9d70 2024-03-29T10:39:13.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.555+0000 7f298effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:13.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.555+0000 7f298effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:13.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.911+0000 7f29a06fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.911+0000 7f29a06fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.911+0000 7f29a06fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.911+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2980005330 con 0x7f29980a9d70 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.912+0000 7f298effd640 1 -- 172.21.15.94:0/1612654508 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f2988052320 con 0x7f29980a9d70 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.912+0000 7f298effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.912+0000 7f298effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:13.896 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 10 librados: watch_flush enter 2024-03-29T10:39:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 10 librados: watch_flush exit 2024-03-29T10:39:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T10:39:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 10 client.4248.objecter successfully canceled tick 2024-03-29T10:39:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 >> v1:172.21.15.94:6816/3829934808 conn(0x7f297c06dd40 legacy=0x7f297c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 10 monclient: shutdown 2024-03-29T10:39:13.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:13.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 >> v1:172.21.15.94:6789/0 conn(0x7f29980a9d70 legacy=0x7f2998141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:13.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 shutdown_connections 2024-03-29T10:39:13.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.915+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 >> 172.21.15.94:0/1612654508 conn(0x7f2998066a00 msgr2=0x7f2998066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:13.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.916+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 shutdown_connections 2024-03-29T10:39:13.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.916+0000 7f29a06fb640 1 -- 172.21.15.94:0/1612654508 wait complete. 2024-03-29T10:39:13.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:13.916+0000 7f29a06fb640 1 librados: shutdown 2024-03-29T10:39:13.914 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:20.722 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.874+0000 7fb30233a640 10 monclient: get_monmap_and_config 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.874+0000 7fb30233a640 10 monclient: build_initial_monmap 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.874+0000 7fb30233a640 10 monclient: monmap: 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:20.874986+0000 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:20.874986+0000 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:20.858 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:20.859 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:20.859 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.874+0000 7fb30233a640 1 Processor -- start 2024-03-29T10:39:20.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.874+0000 7fb30233a640 1 -- start start 2024-03-29T10:39:20.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.874+0000 7fb30233a640 10 monclient: init 2024-03-29T10:39:20.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:20.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:20.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc1457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:20.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 10 monclient(hunting): picked mon.noname-b con 0x7fb2fc13f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:20.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb2fb7fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fb2fc13f650 0x7fb2fc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:37434/0 (socket says 172.21.15.94:37434) 2024-03-29T10:39:20.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb2fb7fe640 1 -- 172.21.15.94:0/3555982829 learned_addr learned my addr 172.21.15.94:0/3555982829 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:20.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0aaef0 con 0x7fb2fc0a9d70 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc145bd0 con 0x7fb2fc1457f0 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.875+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc145eb0 con 0x7fb2fc13f650 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb30233a640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb30233a640 10 monclient(hunting): authenticate will time out at 812.144116s 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545260964 0 0) 0x7fb2fc145eb0 con 0x7fb2fc13f650 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e4003520 con 0x7fb2fc13f650 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2278789451 0 0) 0x7fb2e4003520 con 0x7fb2fc13f650 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 >> v1:172.21.15.173:6789/0 conn(0x7fb2fc1457f0 legacy=0x7fb2fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:20.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.876+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 >> v1:172.21.15.94:6789/0 conn(0x7fb2fc0a9d70 legacy=0x7fb2fc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:20.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:20.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 1 monclient: found mon.noname-b 2024-03-29T10:39:20.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:20.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:39:20.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc146340 con 0x7fb2fc13f650 2024-03-29T10:39:20.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:20.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb2f0001690 con 0x7fb2fc13f650 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.879+0000 7fb2faffd640 10 monclient: dump: 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:20.866 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2f0001860 con 0x7fb2fc13f650 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb2faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb2faffd640 1 -- 172.21.15.94:0/3555982829 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb2f0001cb0 con 0x7fb2fc13f650 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb2faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb2faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb2faffd640 10 monclient: dump: 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:20.867 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb30233a640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb30233a640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.880+0000 7fb30233a640 4 set_mon_vals no callback set 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.881+0000 7fb30233a640 10 monclient: shutdown 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.881+0000 7fb30233a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.881+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 >> v1:172.21.15.94:6790/0 conn(0x7fb2fc13f650 legacy=0x7fb2fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.881+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 shutdown_connections 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.881+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 >> 172.21.15.94:0/3555982829 conn(0x7fb2fc066a00 msgr2=0x7fb2fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.881+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 shutdown_connections 2024-03-29T10:39:20.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.881+0000 7fb30233a640 1 -- 172.21.15.94:0/3555982829 wait complete. 2024-03-29T10:39:20.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.882+0000 7fb30233a640 10 monclient: build_initial_monmap 2024-03-29T10:39:20.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.882+0000 7fb30233a640 10 monclient: monmap: 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.883+0000 7fb30233a640 1 librados: starting msgr at 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.883+0000 7fb30233a640 1 librados: starting objecter 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.883+0000 7fb30233a640 1 Processor -- start 2024-03-29T10:39:20.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.883+0000 7fb30233a640 1 -- start start 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.883+0000 7fb30233a640 1 librados: setting wanted keys 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.883+0000 7fb30233a640 1 librados: calling monclient init 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.883+0000 7fb30233a640 10 monclient: init 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.884+0000 7fb30233a640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.884+0000 7fb30233a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.884+0000 7fb30233a640 10 monclient(hunting): picked mon.noname-b con 0x7fb2fc0a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.884+0000 7fb30233a640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc13f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb2fbfff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fb2fc0a9d70 0x7fb2fc142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45248/0 (socket says 172.21.15.94:45248) 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb2fbfff640 1 -- 172.21.15.94:0/390215331 learned_addr learned my addr 172.21.15.94:0/390215331 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb30233a640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc1457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb2f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb2fc0a9d70 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb2f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc1420a0 con 0x7fb2fc1457f0 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc137a20 con 0x7fb2fc0a9d70 2024-03-29T10:39:20.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0ac5f0 con 0x7fb2fc13f650 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb30233a640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.885+0000 7fb30233a640 10 monclient(hunting): authenticate will time out at 812.153945s 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.886+0000 7fb2f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb2fc13f650 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.886+0000 7fb2f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.886+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229047859 0 0) 0x7fb2fc137a20 con 0x7fb2fc0a9d70 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.886+0000 7fb2f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.886+0000 7fb2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:39:20.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e8004eb0 con 0x7fb2fc0a9d70 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293870493 0 0) 0x7fb2fc0ac5f0 con 0x7fb2fc13f650 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc137a20 con 0x7fb2fc13f650 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1443170100 0 0) 0x7fb2e8004eb0 con 0x7fb2fc0a9d70 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2fc0ac5f0 con 0x7fb2fc0a9d70 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.887+0000 7fb2f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb2fc1457f0 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb2ec001990 con 0x7fb2fc0a9d70 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:20.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4275218273 0 0) 0x7fb2fc137a20 con 0x7fb2fc13f650 2024-03-29T10:39:20.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2e8004eb0 con 0x7fb2fc13f650 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492647194 0 0) 0x7fb2fc1420a0 con 0x7fb2fc1457f0 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc137a20 con 0x7fb2fc1457f0 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb2f0001b70 con 0x7fb2fc13f650 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.880 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277272028 0 0) 0x7fb2fc137a20 con 0x7fb2fc1457f0 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2fc1420a0 con 0x7fb2fc1457f0 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb2f40019f0 con 0x7fb2fc1457f0 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:20.881 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 52878295 0 0) 0x7fb2e8004eb0 con 0x7fb2fc13f650 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 >> v1:172.21.15.173:6789/0 conn(0x7fb2fc0a9d70 legacy=0x7fb2fc142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 >> v1:172.21.15.94:6789/0 conn(0x7fb2fc1457f0 legacy=0x7fb2fc137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 monclient: found mon.c 2024-03-29T10:39:20.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0ac8d0 con 0x7fb2fc13f650 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 10 monclient: _renew_subs 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc142c40 con 0x7fb2fc13f650 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 10 monclient: _renew_subs 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:20.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc13fda0 con 0x7fb2fc13f650 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb30233a640 1 librados: init done 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2f0001cd0 con 0x7fb2fc13f650 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2fa7fc640 4 set_mon_vals no callback set 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb2f0004d90 con 0x7fb2fc13f650 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.888+0000 7fb2f8ff9640 10 monclient: dump: 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:20.884 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.890+0000 7fb30233a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.890+0000 7fb30233a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.890+0000 7fb30233a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:20.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.890+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc005080 con 0x7fb2fc13f650 2024-03-29T10:39:20.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.892+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fb2f001ad60 con 0x7fb2fc13f650 2024-03-29T10:39:20.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.893+0000 7fb2fbfff640 10 monclient: get_auth_request con 0x7fb2e806e220 auth_method 0 2024-03-29T10:39:20.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.893+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fb2f00849a0 con 0x7fb2fc13f650 2024-03-29T10:39:20.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.893+0000 7fb2f8ff9640 10 client.4253.objecter ms_dispatch 0x7fb2fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:20.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.893+0000 7fb2f8ff9640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:20.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.893+0000 7fb2f8ff9640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:20.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.893+0000 7fb2f8ff9640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T10:39:20.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.893+0000 7fb2f8ff9640 10 client.4253.objecter ms_handle_connect 0x7fb2e806e220 2024-03-29T10:39:20.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.900+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2f0050280 con 0x7fb2fc13f650 2024-03-29T10:39:20.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.900+0000 7fb2f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:20.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:20.900+0000 7fb2f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:21.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.217+0000 7fb30233a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:21.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.217+0000 7fb30233a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:21.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.217+0000 7fb30233a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:21.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.217+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2dc005330 con 0x7fb2fc13f650 2024-03-29T10:39:21.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.218+0000 7fb2f8ff9640 1 -- 172.21.15.94:0/390215331 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fb2f00521b0 con 0x7fb2fc13f650 2024-03-29T10:39:21.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.218+0000 7fb2f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:21.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.218+0000 7fb2f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:21.202 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:21.202 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:21.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 10 librados: watch_flush enter 2024-03-29T10:39:21.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 10 librados: watch_flush exit 2024-03-29T10:39:21.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T10:39:21.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 10 client.4253.objecter successfully canceled tick 2024-03-29T10:39:21.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 >> v1:172.21.15.94:6816/3829934808 conn(0x7fb2e806e220 legacy=0x7fb2e80706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 10 monclient: shutdown 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 >> v1:172.21.15.94:6790/0 conn(0x7fb2fc13f650 legacy=0x7fb2fc136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 shutdown_connections 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 >> 172.21.15.94:0/390215331 conn(0x7fb2fc066a00 msgr2=0x7fb2fc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 shutdown_connections 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 1 -- 172.21.15.94:0/390215331 wait complete. 2024-03-29T10:39:21.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:21.221+0000 7fb30233a640 1 librados: shutdown 2024-03-29T10:39:21.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:28.026 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.200+0000 7f6c213e6640 10 monclient: get_monmap_and_config 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.200+0000 7f6c213e6640 10 monclient: build_initial_monmap 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.200+0000 7f6c213e6640 10 monclient: monmap: 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:28.201138+0000 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:28.201138+0000 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:28.184 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:28.185 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:28.185 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:28.185 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:28.185 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:28.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.201+0000 7f6c213e6640 1 Processor -- start 2024-03-29T10:39:28.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.201+0000 7f6c213e6640 1 -- start start 2024-03-29T10:39:28.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.201+0000 7f6c213e6640 10 monclient: init 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c1c06c250 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c1c072400 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c1c06c630 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c06d760 con 0x7f6c1c06c250 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c070500 con 0x7f6c1c072400 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0aff50 con 0x7f6c1c06c630 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:28.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.202+0000 7f6c213e6640 10 monclient(hunting): authenticate will time out at 819.470513s 2024-03-29T10:39:28.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.206+0000 7f6c1a7fc640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f6c1c072400 0x7f6c1c1465b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:43375/0 (socket says 172.21.15.94:43375) 2024-03-29T10:39:28.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.206+0000 7f6c1a7fc640 1 -- 172.21.15.94:0/3742601492 learned_addr learned my addr 172.21.15.94:0/3742601492 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:28.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.207+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842817876 0 0) 0x7f6c1c070500 con 0x7f6c1c072400 2024-03-29T10:39:28.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.207+0000 7f6c19ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:28.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.207+0000 7f6c19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:28.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.207+0000 7f6c19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:28.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.207+0000 7f6c19ffb640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:39:28.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.207+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c08003520 con 0x7f6c1c072400 2024-03-29T10:39:28.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2143581512 0 0) 0x7f6c08003520 con 0x7f6c1c072400 2024-03-29T10:39:28.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 >> v1:172.21.15.94:6790/0 conn(0x7f6c1c06c630 legacy=0x7f6c1c151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:28.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 >> v1:172.21.15.94:6789/0 conn(0x7f6c1c06c250 legacy=0x7f6c1c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 1 monclient: found mon.noname-c 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.208+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0b00e0 con 0x7f6c1c072400 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c19ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c19ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6c14001690 con 0x7f6c1c072400 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:28.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c19ffb640 10 monclient: dump: 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c213e6640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:39:28.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.209+0000 7f6c213e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:28.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.212+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c140017f0 con 0x7f6c1c072400 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.212+0000 7f6c19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.212+0000 7f6c213e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.212+0000 7f6c213e6640 4 set_mon_vals no callback set 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c19ffb640 1 -- 172.21.15.94:0/3742601492 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6c14001c40 con 0x7f6c1c072400 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c19ffb640 10 monclient: dump: 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.198 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c213e6640 10 monclient: shutdown 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c213e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c213e6640 1 -- 172.21.15.94:0/3742601492 >> v1:172.21.15.173:6789/0 conn(0x7f6c1c072400 legacy=0x7f6c1c1465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c213e6640 1 -- 172.21.15.94:0/3742601492 shutdown_connections 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c213e6640 1 -- 172.21.15.94:0/3742601492 >> 172.21.15.94:0/3742601492 conn(0x7f6c1c067610 msgr2=0x7f6c1c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:28.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c213e6640 1 -- 172.21.15.94:0/3742601492 shutdown_connections 2024-03-29T10:39:28.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.213+0000 7f6c213e6640 1 -- 172.21.15.94:0/3742601492 wait complete. 2024-03-29T10:39:28.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.214+0000 7f6c213e6640 10 monclient: build_initial_monmap 2024-03-29T10:39:28.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.214+0000 7f6c213e6640 10 monclient: monmap: 2024-03-29T10:39:28.202 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:28.202 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:28.202 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.214+0000 7f6c213e6640 1 librados: starting msgr at 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.214+0000 7f6c213e6640 1 librados: starting objecter 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.215+0000 7f6c213e6640 1 Processor -- start 2024-03-29T10:39:28.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.215+0000 7f6c213e6640 1 -- start start 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.215+0000 7f6c213e6640 1 librados: setting wanted keys 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.215+0000 7f6c213e6640 1 librados: calling monclient init 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.215+0000 7f6c213e6640 10 monclient: init 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6c1c06c250 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6c1c06c630 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6c1c072400 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0a9930 con 0x7f6c1c06c250 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0aac70 con 0x7f6c1c072400 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c0aaec0 con 0x7f6c1c06c630 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:28.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.216+0000 7f6c213e6640 10 monclient(hunting): authenticate will time out at 819.484535s 2024-03-29T10:39:28.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c1a7fc640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f6c1c06c630 0x7f6c1c0ae920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:33703/0 (socket says 172.21.15.94:33703) 2024-03-29T10:39:28.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c1a7fc640 1 -- 172.21.15.94:0/4173540210 learned_addr learned my addr 172.21.15.94:0/4173540210 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:28.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 10 client.?.objecter ms_handle_connect 0x7f6c1c06c630 2024-03-29T10:39:28.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:28.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287632872 0 0) 0x7f6c1c0aaec0 con 0x7f6c1c06c630 2024-03-29T10:39:28.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:28.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:28.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:28.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:39:28.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.220+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bfc004eb0 con 0x7f6c1c06c630 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 client.?.objecter ms_handle_connect 0x7f6c1c072400 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346273982 0 0) 0x7f6c1c0aac70 con 0x7f6c1c072400 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c1c0aaec0 con 0x7f6c1c072400 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 861251969 0 0) 0x7f6bfc004eb0 con 0x7f6c1c06c630 2024-03-29T10:39:28.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c1c0aac70 con 0x7f6c1c06c630 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6c14001bc0 con 0x7f6c1c06c630 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.224+0000 7f6c137fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 10 monclient(hunting): dump: 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:28.213 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2511528337 0 0) 0x7f6c1c0aac70 con 0x7f6c1c06c630 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 >> v1:172.21.15.173:6789/0 conn(0x7f6c1c072400 legacy=0x7f6c1c1e6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 >> v1:172.21.15.94:6789/0 conn(0x7f6c1c06c250 legacy=0x7f6c1c1db690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 1 monclient: found mon.c 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:28.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0a9dc0 con 0x7f6c1c06c630 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.225+0000 7f6c137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.226+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c14004a90 con 0x7f6c1c06c630 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.226+0000 7f6c137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.226+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6c14004da0 con 0x7f6c1c06c630 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.226+0000 7f6c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.226+0000 7f6c137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.226+0000 7f6c137fe640 10 monclient: dump: 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:28.215 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.226+0000 7f6c197fa640 4 set_mon_vals no callback set 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 5 monclient: authenticate success, global_id 4259 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 10 monclient: _renew_subs 2024-03-29T10:39:28.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:28.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c0a94e0 con 0x7f6c1c06c630 2024-03-29T10:39:28.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:28.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 10 monclient: _renew_subs 2024-03-29T10:39:28.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:28.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c1dbf80 con 0x7f6c1c06c630 2024-03-29T10:39:28.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.227+0000 7f6c213e6640 1 librados: init done 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.229+0000 7f6c213e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.229+0000 7f6c213e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.230+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f6c1401ad70 con 0x7f6c1c06c630 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.230+0000 7f6c213e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.230+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c0a9710 con 0x7f6c1c06c630 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.231+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f6c14084ab0 con 0x7f6c1c06c630 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.231+0000 7f6c137fe640 10 client.4259.objecter ms_dispatch 0x7f6c1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.231+0000 7f6c137fe640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.231+0000 7f6c1affd640 10 monclient: get_auth_request con 0x7f6bfc06dc40 auth_method 0 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.231+0000 7f6c137fe640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:28.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.231+0000 7f6c137fe640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T10:39:28.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.231+0000 7f6c137fe640 10 client.4259.objecter ms_handle_connect 0x7f6bfc06dc40 2024-03-29T10:39:28.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.240+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c14050390 con 0x7f6c1c06c630 2024-03-29T10:39:28.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.240+0000 7f6c137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:28.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.240+0000 7f6c137fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:28.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.574+0000 7f6c213e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:28.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.574+0000 7f6c213e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:28.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.574+0000 7f6c213e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:28.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.574+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c1c075f10 con 0x7f6c1c06c630 2024-03-29T10:39:28.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.575+0000 7f6c137fe640 1 -- 172.21.15.94:0/4173540210 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f6c140522c0 con 0x7f6c1c06c630 2024-03-29T10:39:28.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.575+0000 7f6c137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:28.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.575+0000 7f6c137fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:28.561 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:28.561 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:28.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 10 librados: watch_flush enter 2024-03-29T10:39:28.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 10 librados: watch_flush exit 2024-03-29T10:39:28.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T10:39:28.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 10 client.4259.objecter successfully canceled tick 2024-03-29T10:39:28.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 >> v1:172.21.15.94:6816/3829934808 conn(0x7f6bfc06dc40 legacy=0x7f6bfc070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:28.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 10 monclient: shutdown 2024-03-29T10:39:28.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:28.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 >> v1:172.21.15.94:6790/0 conn(0x7f6c1c06c630 legacy=0x7f6c1c0ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:28.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 shutdown_connections 2024-03-29T10:39:28.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.581+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 >> 172.21.15.94:0/4173540210 conn(0x7f6c1c067610 msgr2=0x7f6c1c070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:28.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.583+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 shutdown_connections 2024-03-29T10:39:28.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.583+0000 7f6c213e6640 1 -- 172.21.15.94:0/4173540210 wait complete. 2024-03-29T10:39:28.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:28.583+0000 7f6c213e6640 1 librados: shutdown 2024-03-29T10:39:28.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:35.394 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:35.548 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.563+0000 7f0580dc4640 10 monclient: get_monmap_and_config 2024-03-29T10:39:35.548 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.563+0000 7f0580dc4640 10 monclient: build_initial_monmap 2024-03-29T10:39:35.548 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.563+0000 7f0580dc4640 10 monclient: monmap: 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:35.565671+0000 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:35.565671+0000 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:35.549 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:35.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.564+0000 7f0580dc4640 1 Processor -- start 2024-03-29T10:39:35.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.564+0000 7f0580dc4640 1 -- start start 2024-03-29T10:39:35.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.564+0000 7f0580dc4640 10 monclient: init 2024-03-29T10:39:35.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:35.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f057c1457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f057c13f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f057c0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c0aaef0 con 0x7f057c0a9d70 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c145bd0 con 0x7f057c13f650 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c145eb0 con 0x7f057c1457f0 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:35.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.565+0000 7f0580dc4640 10 monclient(hunting): authenticate will time out at 826.834995s 2024-03-29T10:39:35.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f0579d9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f057c13f650 0x7f057c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:43571/0 (socket says 172.21.15.94:43571) 2024-03-29T10:39:35.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f0579d9b640 1 -- 172.21.15.94:0/607781560 learned_addr learned my addr 172.21.15.94:0/607781560 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:35.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051042514 0 0) 0x7f057c145bd0 con 0x7f057c13f650 2024-03-29T10:39:35.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f057959a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f057959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f057959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f057959a640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:39:35.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.571+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0564003520 con 0x7f057c13f650 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236284029 0 0) 0x7f0564003520 con 0x7f057c13f650 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 >> v1:172.21.15.94:6790/0 conn(0x7f057c1457f0 legacy=0x7f057c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 >> v1:172.21.15.94:6789/0 conn(0x7f057c0a9d70 legacy=0x7f057c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 10 monclient: _finish_hunting 0 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 1 monclient: found mon.noname-c 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c146340 con 0x7f057c13f650 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 10 monclient: _finish_auth 0 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 10 monclient: _check_auth_tickets 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0574001a10 con 0x7f057c13f650 2024-03-29T10:39:35.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f057959a640 10 monclient: dump: 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.560 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:35.561 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:35.561 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:35.561 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:35.561 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:35.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f0580dc4640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:39:35.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.572+0000 7f0580dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:35.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.573+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0574001be0 con 0x7f057c13f650 2024-03-29T10:39:35.562 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.573+0000 7f057959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:35.562 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.573+0000 7f057959a640 1 -- 172.21.15.94:0/607781560 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0574002030 con 0x7f057c13f650 2024-03-29T10:39:35.562 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.573+0000 7f057959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.562 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.573+0000 7f057959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:35.562 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.573+0000 7f057959a640 10 monclient: dump: 2024-03-29T10:39:35.562 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:35.562 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:35.563 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.563 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.563 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.563 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:35.563 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:35.563 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:35.564 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:35.564 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:35.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.574+0000 7f0580dc4640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:35.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.574+0000 7f0580dc4640 4 set_mon_vals no callback set 2024-03-29T10:39:35.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.574+0000 7f0580dc4640 10 monclient: shutdown 2024-03-29T10:39:35.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.574+0000 7f0580dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:35.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.574+0000 7f0580dc4640 1 -- 172.21.15.94:0/607781560 >> v1:172.21.15.173:6789/0 conn(0x7f057c13f650 legacy=0x7f057c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.576+0000 7f0580dc4640 1 -- 172.21.15.94:0/607781560 shutdown_connections 2024-03-29T10:39:35.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.576+0000 7f0580dc4640 1 -- 172.21.15.94:0/607781560 >> 172.21.15.94:0/607781560 conn(0x7f057c066a00 msgr2=0x7f057c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:35.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.578+0000 7f0580dc4640 1 -- 172.21.15.94:0/607781560 shutdown_connections 2024-03-29T10:39:35.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.579+0000 7f0580dc4640 1 -- 172.21.15.94:0/607781560 wait complete. 2024-03-29T10:39:35.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.580+0000 7f0580dc4640 10 monclient: build_initial_monmap 2024-03-29T10:39:35.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.580+0000 7f0580dc4640 10 monclient: monmap: 2024-03-29T10:39:35.569 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:35.569 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:35.570 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:35.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.580+0000 7f0580dc4640 1 librados: starting msgr at 2024-03-29T10:39:35.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.580+0000 7f0580dc4640 1 librados: starting objecter 2024-03-29T10:39:35.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.580+0000 7f0580dc4640 1 Processor -- start 2024-03-29T10:39:35.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.581+0000 7f0580dc4640 1 -- start start 2024-03-29T10:39:35.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.581+0000 7f0580dc4640 1 librados: setting wanted keys 2024-03-29T10:39:35.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.581+0000 7f0580dc4640 1 librados: calling monclient init 2024-03-29T10:39:35.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.581+0000 7f0580dc4640 10 monclient: init 2024-03-29T10:39:35.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.582+0000 7f0580dc4640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:35.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.582+0000 7f0580dc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:35.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.582+0000 7f0580dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f057c0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:35.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0580dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f057c13f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:35.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0579d9b640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f057c13f650 0x7f057c1e7920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:46879/0 (socket says 172.21.15.94:46879) 2024-03-29T10:39:35.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0580dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f057c1457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:35.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0580dc4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c06fb90 con 0x7f057c0a9d70 2024-03-29T10:39:35.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0580dc4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c06e370 con 0x7f057c1457f0 2024-03-29T10:39:35.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0580dc4640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c13fda0 con 0x7f057c13f650 2024-03-29T10:39:35.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0580dc4640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:35.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f0580dc4640 10 monclient(hunting): authenticate will time out at 826.854678s 2024-03-29T10:39:35.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f057a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f057c0a9d70 0x7f057c142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59735/0 (socket says 172.21.15.94:59735) 2024-03-29T10:39:35.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.585+0000 7f057a59c640 1 -- 172.21.15.94:0/1459752045 learned_addr learned my addr 172.21.15.94:0/1459752045 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:35.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 10 client.?.objecter ms_handle_connect 0x7f057c0a9d70 2024-03-29T10:39:35.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:35.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192058235 0 0) 0x7f057c06fb90 con 0x7f057c0a9d70 2024-03-29T10:39:35.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f055c004eb0 con 0x7f057c0a9d70 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.587+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3253388569 0 0) 0x7f055c004eb0 con 0x7f057c0a9d70 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f057c06fb90 con 0x7f057c0a9d70 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f05700017d0 con 0x7f057c0a9d70 2024-03-29T10:39:35.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient(hunting): dump: 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:35.585 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1397755884 0 0) 0x7f057c06fb90 con 0x7f057c0a9d70 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 >> v1:172.21.15.94:6790/0 conn(0x7f057c13f650 legacy=0x7f057c1e7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 >> v1:172.21.15.173:6789/0 conn(0x7f057c1457f0 legacy=0x7f057c1ee920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 1 monclient: found mon.a 2024-03-29T10:39:35.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c13ff30 con 0x7f057c0a9d70 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient: _finish_auth 0 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.588+0000 7f056affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f0580dc4640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f0580dc4640 10 monclient: _renew_subs 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0570004860 con 0x7f057c0a9d70 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f0580dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c1e8030 con 0x7f057c0a9d70 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f056affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f05700017d0 con 0x7f057c0a9d70 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f056affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f056affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:35.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.589+0000 7f056affd640 10 monclient: dump: 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.590+0000 7f0580dc4640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.590+0000 7f0580dc4640 10 monclient: _renew_subs 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.590+0000 7f0580dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:35.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.590+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c06e850 con 0x7f057c0a9d70 2024-03-29T10:39:35.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.590+0000 7f0580dc4640 1 librados: init done 2024-03-29T10:39:35.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.593+0000 7f056bfff640 4 set_mon_vals no callback set 2024-03-29T10:39:35.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.595+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f057001a840 con 0x7f057c0a9d70 2024-03-29T10:39:35.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.595+0000 7f0580dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.595+0000 7f0580dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.595+0000 7f0580dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:35.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.595+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0560005080 con 0x7f057c0a9d70 2024-03-29T10:39:35.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.596+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0570084790 con 0x7f057c0a9d70 2024-03-29T10:39:35.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.596+0000 7f056affd640 10 client.4257.objecter ms_dispatch 0x7f057c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:35.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.596+0000 7f056affd640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:35.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.596+0000 7f056affd640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:35.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.596+0000 7f056affd640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T10:39:35.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.596+0000 7f0579d9b640 10 monclient: get_auth_request con 0x7f055c06dd60 auth_method 0 2024-03-29T10:39:35.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.597+0000 7f056affd640 10 client.4257.objecter ms_handle_connect 0x7f055c06dd60 2024-03-29T10:39:35.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.606+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05700500d0 con 0x7f057c0a9d70 2024-03-29T10:39:35.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.606+0000 7f056affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.606+0000 7f056affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:35.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.936+0000 7f0580dc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.937+0000 7f0580dc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.937+0000 7f0580dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.937+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0560005330 con 0x7f057c0a9d70 2024-03-29T10:39:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.938+0000 7f056affd640 1 -- 172.21.15.94:0/1459752045 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f05700500d0 con 0x7f057c0a9d70 2024-03-29T10:39:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.938+0000 7f056affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.938+0000 7f056affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:35.923 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:35.923 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:35.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.943+0000 7f0580dc4640 10 librados: watch_flush enter 2024-03-29T10:39:35.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.944+0000 7f0580dc4640 10 librados: watch_flush exit 2024-03-29T10:39:35.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.944+0000 7f0580dc4640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T10:39:35.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.944+0000 7f0580dc4640 10 client.4257.objecter successfully canceled tick 2024-03-29T10:39:35.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.944+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 >> v1:172.21.15.94:6816/3829934808 conn(0x7f055c06dd60 legacy=0x7f055c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.944+0000 7f0580dc4640 10 monclient: shutdown 2024-03-29T10:39:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.944+0000 7f0580dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.944+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 >> v1:172.21.15.94:6789/0 conn(0x7f057c0a9d70 legacy=0x7f057c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.945+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 shutdown_connections 2024-03-29T10:39:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.945+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 >> 172.21.15.94:0/1459752045 conn(0x7f057c066a00 msgr2=0x7f057c139020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.945+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 shutdown_connections 2024-03-29T10:39:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.945+0000 7f0580dc4640 1 -- 172.21.15.94:0/1459752045 wait complete. 2024-03-29T10:39:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:35.945+0000 7f0580dc4640 1 librados: shutdown 2024-03-29T10:39:35.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:42.754 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:42.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.918+0000 7f1cf14c1640 10 monclient: get_monmap_and_config 2024-03-29T10:39:42.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.918+0000 7f1cf14c1640 10 monclient: build_initial_monmap 2024-03-29T10:39:42.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.918+0000 7f1cf14c1640 10 monclient: monmap: 2024-03-29T10:39:42.905 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:42.905 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:42.905 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:42.920797+0000 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:42.920797+0000 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.920+0000 7f1cf14c1640 1 Processor -- start 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.920+0000 7f1cf14c1640 1 -- start start 2024-03-29T10:39:42.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.920+0000 7f1cf14c1640 10 monclient: init 2024-03-29T10:39:42.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:42.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:42.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec0a9930 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:42.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec149dc0 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec14ff40 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0aae60 con 0x7f1cec149dc0 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0b4310 con 0x7f1cec14ff40 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0afdd0 con 0x7f1cec0a9930 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.921+0000 7f1cf14c1640 10 monclient(hunting): authenticate will time out at 834.190307s 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.924+0000 7f1cea7fc640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1cec149dc0 0x7f1cec14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:46905/0 (socket says 172.21.15.94:46905) 2024-03-29T10:39:42.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.924+0000 7f1cea7fc640 1 -- 172.21.15.94:0/2178237571 learned_addr learned my addr 172.21.15.94:0/2178237571 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:42.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.928+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483781401 0 0) 0x7f1cec0aae60 con 0x7f1cec149dc0 2024-03-29T10:39:42.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.928+0000 7f1ce9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:42.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.928+0000 7f1ce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:42.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.928+0000 7f1ce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:42.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.928+0000 7f1ce9ffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:39:42.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.928+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4003520 con 0x7f1cec149dc0 2024-03-29T10:39:42.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319370225 0 0) 0x7f1cec0afdd0 con 0x7f1cec0a9930 2024-03-29T10:39:42.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:42.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:42.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:42.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:39:42.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cec0aae60 con 0x7f1cec0a9930 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218650478 0 0) 0x7f1cd4003520 con 0x7f1cec149dc0 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 >> v1:172.21.15.94:6790/0 conn(0x7f1cec0a9930 legacy=0x7f1cec143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 >> v1:172.21.15.173:6789/0 conn(0x7f1cec14ff40 legacy=0x7f1cec152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 monclient: found mon.noname-a 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec14a950 con 0x7f1cec149dc0 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1ce4001990 con 0x7f1cec149dc0 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1cf14c1640 5 monclient: authenticate success, global_id 4260 2024-03-29T10:39:42.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1cf14c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: dump: 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:42.917 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce4001bb0 con 0x7f1cec149dc0 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 1 -- 172.21.15.94:0/2178237571 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1ce4001f40 con 0x7f1cec149dc0 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.929+0000 7f1ce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.930+0000 7f1ce9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.930+0000 7f1ce9ffb640 10 monclient: dump: 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:42.918 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.930+0000 7f1cf14c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.930+0000 7f1cf14c1640 4 set_mon_vals no callback set 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.930+0000 7f1cf14c1640 10 monclient: shutdown 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.930+0000 7f1cf14c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:42.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.930+0000 7f1cf14c1640 1 -- 172.21.15.94:0/2178237571 >> v1:172.21.15.94:6789/0 conn(0x7f1cec149dc0 legacy=0x7f1cec14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:42.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.932+0000 7f1cf14c1640 1 -- 172.21.15.94:0/2178237571 shutdown_connections 2024-03-29T10:39:42.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.932+0000 7f1cf14c1640 1 -- 172.21.15.94:0/2178237571 >> 172.21.15.94:0/2178237571 conn(0x7f1cec0b35e0 msgr2=0x7f1cec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.932+0000 7f1cf14c1640 1 -- 172.21.15.94:0/2178237571 shutdown_connections 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.932+0000 7f1cf14c1640 1 -- 172.21.15.94:0/2178237571 wait complete. 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.933+0000 7f1cf14c1640 10 monclient: build_initial_monmap 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.933+0000 7f1cf14c1640 10 monclient: monmap: 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:42.921 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.933+0000 7f1cf14c1640 1 librados: starting msgr at 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.933+0000 7f1cf14c1640 1 librados: starting objecter 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.934+0000 7f1cf14c1640 1 Processor -- start 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.934+0000 7f1cf14c1640 1 -- start start 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.934+0000 7f1cf14c1640 1 librados: setting wanted keys 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.934+0000 7f1cf14c1640 1 librados: calling monclient init 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.934+0000 7f1cf14c1640 10 monclient: init 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:42.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 10 monclient(hunting): picked mon.noname-a con 0x7f1cec0a9930 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:42.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 10 monclient(hunting): picked mon.noname-c con 0x7f1cec14ff40 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:42.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 10 monclient(hunting): picked mon.noname-b con 0x7f1cec1e7b50 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:42.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec13bb60 con 0x7f1cec0a9930 2024-03-29T10:39:42.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec13fab0 con 0x7f1cec1e7b50 2024-03-29T10:39:42.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec13ffd0 con 0x7f1cec14ff40 2024-03-29T10:39:42.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:42.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.935+0000 7f1cf14c1640 10 monclient(hunting): authenticate will time out at 834.204245s 2024-03-29T10:39:42.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cea7fc640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f1cec14ff40 0x7f1cec1e7440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:43283/0 (socket says 172.21.15.94:43283) 2024-03-29T10:39:42.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cea7fc640 1 -- 172.21.15.94:0/159573185 learned_addr learned my addr 172.21.15.94:0/159573185 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:42.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 10 client.?.objecter ms_handle_connect 0x7f1cec14ff40 2024-03-29T10:39:42.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:42.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400902048 0 0) 0x7f1cec13ffd0 con 0x7f1cec14ff40 2024-03-29T10:39:42.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:42.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.938+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd8004eb0 con 0x7f1cec14ff40 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3994555373 0 0) 0x7f1cd8004eb0 con 0x7f1cec14ff40 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cec13ffd0 con 0x7f1cec14ff40 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1ce40018d0 con 0x7f1cec14ff40 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 10 monclient(hunting): dump: 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.926 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1785710251 0 0) 0x7f1cec13ffd0 con 0x7f1cec14ff40 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 >> v1:172.21.15.173:6789/0 conn(0x7f1cec1e7b50 legacy=0x7f1cec1ee980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:39:42.927 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.939+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 >> v1:172.21.15.94:6789/0 conn(0x7f1cec0a9930 legacy=0x7f1cec13ea60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:42.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:42.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 1 monclient: found mon.c 2024-03-29T10:39:42.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:42.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec13f390 con 0x7f1cec14ff40 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 5 monclient: authenticate success, global_id 4265 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 10 monclient: _renew_subs 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec13f640 con 0x7f1cec14ff40 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 10 monclient: _renew_subs 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec1f0280 con 0x7f1cec14ff40 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.942+0000 7f1cf14c1640 1 librados: init done 2024-03-29T10:39:42.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cf14c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cf14c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cf14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce4004e10 con 0x7f1cec14ff40 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec13fca0 con 0x7f1cec14ff40 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cd37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1ce4005120 con 0x7f1cec14ff40 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1cd37fe640 10 monclient: dump: 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:42.930 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:42.931 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:42.931 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:42.931 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:42.931 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:42.931 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:42.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.943+0000 7f1ce8ff9640 4 set_mon_vals no callback set 2024-03-29T10:39:42.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.945+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f1ce401b0f0 con 0x7f1cec14ff40 2024-03-29T10:39:42.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.946+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f1ce4054be0 con 0x7f1cec14ff40 2024-03-29T10:39:42.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.946+0000 7f1cd37fe640 10 client.4265.objecter ms_dispatch 0x7f1cec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:42.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.946+0000 7f1cd37fe640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:42.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.946+0000 7f1cd37fe640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:42.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.946+0000 7f1cd37fe640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T10:39:42.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.948+0000 7f1ceaffd640 10 monclient: get_auth_request con 0x7f1cd806dc50 auth_method 0 2024-03-29T10:39:42.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.949+0000 7f1cd37fe640 10 client.4265.objecter ms_handle_connect 0x7f1cd806dc50 2024-03-29T10:39:42.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.956+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ce4050e80 con 0x7f1cec14ff40 2024-03-29T10:39:42.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.957+0000 7f1cd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:42.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:42.957+0000 7f1cd37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.286+0000 7f1cf14c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.286+0000 7f1cf14c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.286+0000 7f1cf14c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.286+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cec075f10 con 0x7f1cec14ff40 2024-03-29T10:39:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.289+0000 7f1cd37fe640 1 -- 172.21.15.94:0/159573185 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f1ce4051020 con 0x7f1cec14ff40 2024-03-29T10:39:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.289+0000 7f1cd37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.289+0000 7f1cd37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:43.276 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:43.276 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:43.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 10 librados: watch_flush enter 2024-03-29T10:39:43.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 10 librados: watch_flush exit 2024-03-29T10:39:43.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T10:39:43.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 10 client.4265.objecter successfully canceled tick 2024-03-29T10:39:43.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 >> v1:172.21.15.94:6816/3829934808 conn(0x7f1cd806dc50 legacy=0x7f1cd8070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:43.280 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 10 monclient: shutdown 2024-03-29T10:39:43.280 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:43.280 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.294+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 >> v1:172.21.15.94:6790/0 conn(0x7f1cec14ff40 legacy=0x7f1cec1e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:43.283 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.298+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 shutdown_connections 2024-03-29T10:39:43.283 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.298+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 >> 172.21.15.94:0/159573185 conn(0x7f1cec0b35e0 msgr2=0x7f1cec0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:43.283 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.298+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 shutdown_connections 2024-03-29T10:39:43.283 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.298+0000 7f1cf14c1640 1 -- 172.21.15.94:0/159573185 wait complete. 2024-03-29T10:39:43.283 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:43.299+0000 7f1cf14c1640 1 librados: shutdown 2024-03-29T10:39:43.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:50.110 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:50.261 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.277+0000 7f7a5ea33640 10 monclient: get_monmap_and_config 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.277+0000 7f7a5ea33640 10 monclient: build_initial_monmap 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.277+0000 7f7a5ea33640 10 monclient: monmap: 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:50.278373+0000 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:50.278373+0000 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:50.262 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:50.263 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.278+0000 7f7a5ea33640 1 Processor -- start 2024-03-29T10:39:50.263 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.278+0000 7f7a5ea33640 1 -- start start 2024-03-29T10:39:50.263 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.278+0000 7f7a5ea33640 10 monclient: init 2024-03-29T10:39:50.263 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:50.263 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:50.263 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 10 monclient(hunting): picked mon.noname-a con 0x7f7a580a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:50.263 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5813ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:50.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 10 monclient(hunting): picked mon.noname-c con 0x7f7a580af900 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:50.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a580afd10 con 0x7f7a580a9ac0 2024-03-29T10:39:50.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a580af6f0 con 0x7f7a580af900 2024-03-29T10:39:50.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a580aac80 con 0x7f7a5813ee90 2024-03-29T10:39:50.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:50.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.279+0000 7f7a5ea33640 10 monclient(hunting): authenticate will time out at 841.547745s 2024-03-29T10:39:50.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.282+0000 7f7a5cfd0640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f7a580af900 0x7f7a58149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:44731/0 (socket says 172.21.15.94:44731) 2024-03-29T10:39:50.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.282+0000 7f7a5cfd0640 1 -- 172.21.15.94:0/417713416 learned_addr learned my addr 172.21.15.94:0/417713416 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763675946 0 0) 0x7f7a580af6f0 con 0x7f7a580af900 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a40003520 con 0x7f7a580af900 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745667970 0 0) 0x7f7a580aac80 con 0x7f7a5813ee90 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:39:50.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.283+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a580af6f0 con 0x7f7a5813ee90 2024-03-29T10:39:50.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530694147 0 0) 0x7f7a580afd10 con 0x7f7a580a9ac0 2024-03-29T10:39:50.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:50.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:50.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:50.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a580aac80 con 0x7f7a580a9ac0 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3498491367 0 0) 0x7f7a40003520 con 0x7f7a580af900 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 >> v1:172.21.15.94:6790/0 conn(0x7f7a5813ee90 legacy=0x7f7a5813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 >> v1:172.21.15.94:6789/0 conn(0x7f7a580a9ac0 legacy=0x7f7a580adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 monclient: found mon.noname-c 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a580aaf60 con 0x7f7a580af900 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a48004c90 con 0x7f7a580af900 2024-03-29T10:39:50.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: dump: 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a5ea33640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a5ea33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:50.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a48004eb0 con 0x7f7a580af900 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a5ea33640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.286+0000 7f7a5ea33640 4 set_mon_vals no callback set 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.287+0000 7f7a56ffd640 1 -- 172.21.15.94:0/417713416 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a48001410 con 0x7f7a580af900 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.287+0000 7f7a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.287+0000 7f7a56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.287+0000 7f7a56ffd640 10 monclient: dump: 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:50.275 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:50.276 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:50.276 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:50.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.287+0000 7f7a5ea33640 10 monclient: shutdown 2024-03-29T10:39:50.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.287+0000 7f7a5ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:50.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.287+0000 7f7a5ea33640 1 -- 172.21.15.94:0/417713416 >> v1:172.21.15.173:6789/0 conn(0x7f7a580af900 legacy=0x7f7a58149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:50.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.289+0000 7f7a5ea33640 1 -- 172.21.15.94:0/417713416 shutdown_connections 2024-03-29T10:39:50.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.289+0000 7f7a5ea33640 1 -- 172.21.15.94:0/417713416 >> 172.21.15.94:0/417713416 conn(0x7f7a580b35e0 msgr2=0x7f7a580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:50.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.290+0000 7f7a5ea33640 1 -- 172.21.15.94:0/417713416 shutdown_connections 2024-03-29T10:39:50.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.291+0000 7f7a5ea33640 1 -- 172.21.15.94:0/417713416 wait complete. 2024-03-29T10:39:50.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.292+0000 7f7a5ea33640 10 monclient: build_initial_monmap 2024-03-29T10:39:50.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.292+0000 7f7a5ea33640 10 monclient: monmap: 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.292+0000 7f7a5ea33640 1 librados: starting msgr at 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.292+0000 7f7a5ea33640 1 librados: starting objecter 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.293+0000 7f7a5ea33640 1 Processor -- start 2024-03-29T10:39:50.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.293+0000 7f7a5ea33640 1 -- start start 2024-03-29T10:39:50.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.293+0000 7f7a5ea33640 1 librados: setting wanted keys 2024-03-29T10:39:50.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.293+0000 7f7a5ea33640 1 librados: calling monclient init 2024-03-29T10:39:50.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.293+0000 7f7a5ea33640 10 monclient: init 2024-03-29T10:39:50.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.294+0000 7f7a5ea33640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:50.280 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.295+0000 7f7a5ea33640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 10 monclient(hunting): picked mon.noname-a con 0x7f7a580a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 10 monclient(hunting): picked mon.noname-b con 0x7f7a580af900 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5813ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5813fcd0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a581d3a00 con 0x7f7a580af900 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a581d3b90 con 0x7f7a5813ee90 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:50.281 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.297+0000 7f7a5ea33640 10 monclient(hunting): authenticate will time out at 841.565626s 2024-03-29T10:39:50.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.301+0000 7f7a5cfd0640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7a5813ee90 0x7f7a581de650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:33911/0 (socket says 172.21.15.94:33911) 2024-03-29T10:39:50.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.301+0000 7f7a5cfd0640 1 -- 172.21.15.94:0/3181325331 learned_addr learned my addr 172.21.15.94:0/3181325331 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:50.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.304+0000 7f7a54ff9640 10 client.?.objecter ms_handle_connect 0x7f7a580a9ac0 2024-03-29T10:39:50.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.304+0000 7f7a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:50.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.304+0000 7f7a54ff9640 10 client.?.objecter ms_handle_connect 0x7f7a5813ee90 2024-03-29T10:39:50.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.304+0000 7f7a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:50.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859062899 0 0) 0x7f7a5813fcd0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a44004eb0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3349231153 0 0) 0x7f7a44004eb0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a5813fcd0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a50001cf0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.292 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1304854364 0 0) 0x7f7a5813fcd0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 >> v1:172.21.15.94:6790/0 conn(0x7f7a5813ee90 legacy=0x7f7a581de650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 >> v1:172.21.15.173:6789/0 conn(0x7f7a580af900 legacy=0x7f7a581387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:50.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 monclient: found mon.a 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a581d3d20 con 0x7f7a580a9ac0 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient: _finish_auth 0 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a5ea33640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a5ea33640 10 monclient: _renew_subs 2024-03-29T10:39:50.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a5ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.305+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a58138ef0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.306+0000 7f7a5ea33640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.306+0000 7f7a5ea33640 10 monclient: _renew_subs 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.306+0000 7f7a5ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.306+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a58071d30 con 0x7f7a580a9ac0 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.306+0000 7f7a5ea33640 1 librados: init done 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.307+0000 7f7a5ea33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.307+0000 7f7a5ea33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.307+0000 7f7a5ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:50.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.307+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5806f550 con 0x7f7a580a9ac0 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.309+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a50001fd0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.309+0000 7f7a54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.309+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a50003de0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.309+0000 7f7a54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.309+0000 7f7a54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.309+0000 7f7a54ff9640 10 monclient: dump: 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:50.297 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.309+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f7a50019db0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.310+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f7a50083d80 con 0x7f7a580a9ac0 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.310+0000 7f7a54ff9640 10 client.4266.objecter ms_dispatch 0x7f7a58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.310+0000 7f7a54ff9640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.310+0000 7f7a54ff9640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:50.298 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.310+0000 7f7a54ff9640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T10:39:50.299 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.311+0000 7f7a567fc640 4 set_mon_vals no callback set 2024-03-29T10:39:50.299 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.314+0000 7f7a577fe640 10 monclient: get_auth_request con 0x7f7a4406dd90 auth_method 0 2024-03-29T10:39:50.299 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.314+0000 7f7a54ff9640 10 client.4266.objecter ms_handle_connect 0x7f7a4406dd90 2024-03-29T10:39:50.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.316+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a5004f6e0 con 0x7f7a580a9ac0 2024-03-29T10:39:50.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.317+0000 7f7a54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:50.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.317+0000 7f7a54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:50.618 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.634+0000 7f7a5ea33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:50.618 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.634+0000 7f7a5ea33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:50.618 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.634+0000 7f7a5ea33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:39:50.619 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.634+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a58075f10 con 0x7f7a580a9ac0 2024-03-29T10:39:50.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.636+0000 7f7a54ff9640 1 -- 172.21.15.94:0/3181325331 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f7a50051610 con 0x7f7a580a9ac0 2024-03-29T10:39:50.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.636+0000 7f7a54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:50.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.636+0000 7f7a54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:50.621 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:50.621 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 10 librados: watch_flush enter 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 10 librados: watch_flush exit 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 10 client.4266.objecter successfully canceled tick 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 >> v1:172.21.15.94:6816/3829934808 conn(0x7f7a4406dd90 legacy=0x7f7a44070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 10 monclient: shutdown 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 >> v1:172.21.15.94:6789/0 conn(0x7f7a580a9ac0 legacy=0x7f7a5813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:50.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 shutdown_connections 2024-03-29T10:39:50.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.640+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 >> 172.21.15.94:0/3181325331 conn(0x7f7a580b35e0 msgr2=0x7f7a5813b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:50.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.642+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 shutdown_connections 2024-03-29T10:39:50.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.642+0000 7f7a5ea33640 1 -- 172.21.15.94:0/3181325331 wait complete. 2024-03-29T10:39:50.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:50.643+0000 7f7a5ea33640 1 librados: shutdown 2024-03-29T10:39:50.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:57.453 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:57.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.625+0000 7f4de596d640 10 monclient: get_monmap_and_config 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.625+0000 7f4de596d640 10 monclient: build_initial_monmap 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.625+0000 7f4de596d640 10 monclient: monmap: 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:39:57.626757+0000 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:39:57.626757+0000 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:39:57.610 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:57.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.627+0000 7f4de596d640 1 Processor -- start 2024-03-29T10:39:57.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.630+0000 7f4de596d640 1 -- start start 2024-03-29T10:39:57.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.630+0000 7f4de596d640 10 monclient: init 2024-03-29T10:39:57.621 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:57.621 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 10 monclient(hunting): picked mon.noname-b con 0x7f4dd8006a70 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 10 monclient(hunting): picked mon.noname-a con 0x7f4dd802b100 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 10 monclient(hunting): picked mon.noname-c con 0x7f4dd8029c30 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd802a270 con 0x7f4dd802b100 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd802c440 con 0x7f4dd8029c30 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd80c9440 con 0x7f4dd8006a70 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.631+0000 7f4de596d640 10 monclient(hunting): authenticate will time out at 848.899247s 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.633+0000 7f4de516c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f4dd8029c30 0x7f4dd80d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:55669/0 (socket says 172.21.15.94:55669) 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.634+0000 7f4de516c640 1 -- 172.21.15.94:0/1088217738 learned_addr learned my addr 172.21.15.94:0/1088217738 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.634+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020355222 0 0) 0x7f4dd802a270 con 0x7f4dd802b100 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.634+0000 7f4ddf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:57.622 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.634+0000 7f4ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.634+0000 7f4ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.634+0000 7f4ddf7fe640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.635+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0003520 con 0x7f4dd802b100 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.635+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538024902 0 0) 0x7f4dd802c440 con 0x7f4dd8029c30 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.635+0000 7f4ddf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.635+0000 7f4ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.635+0000 7f4ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.635+0000 7f4ddf7fe640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.635+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd00048a0 con 0x7f4dd8029c30 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364159521 0 0) 0x7f4dd80c9440 con 0x7f4dd8006a70 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:57.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0004be0 con 0x7f4dd8006a70 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659235964 0 0) 0x7f4dd0003520 con 0x7f4dd802b100 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 >> v1:172.21.15.94:6790/0 conn(0x7f4dd8006a70 legacy=0x7f4dd80c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 >> v1:172.21.15.173:6789/0 conn(0x7f4dd8029c30 legacy=0x7f4dd80d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 monclient: found mon.noname-a 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd80c95d0 con 0x7f4dd802b100 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4dcc0015f0 con 0x7f4dd802b100 2024-03-29T10:39:57.624 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: dump: 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4de596d640 5 monclient: authenticate success, global_id 4269 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4de596d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:57.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dcc001750 con 0x7f4dd802b100 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 1 -- 172.21.15.94:0/1088217738 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4dcc001ba0 con 0x7f4dd802b100 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4ddf7fe640 10 monclient: dump: 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:57.626 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4de596d640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.636+0000 7f4de596d640 4 set_mon_vals no callback set 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.637+0000 7f4de596d640 10 monclient: shutdown 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.637+0000 7f4de596d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.638+0000 7f4de596d640 1 -- 172.21.15.94:0/1088217738 >> v1:172.21.15.94:6789/0 conn(0x7f4dd802b100 legacy=0x7f4dd80cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.638+0000 7f4de596d640 1 -- 172.21.15.94:0/1088217738 shutdown_connections 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.638+0000 7f4de596d640 1 -- 172.21.15.94:0/1088217738 >> 172.21.15.94:0/1088217738 conn(0x7f4dd80058f0 msgr2=0x7f4dd8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.638+0000 7f4de596d640 1 -- 172.21.15.94:0/1088217738 shutdown_connections 2024-03-29T10:39:57.627 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.638+0000 7f4de596d640 1 -- 172.21.15.94:0/1088217738 wait complete. 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.639+0000 7f4de596d640 10 monclient: build_initial_monmap 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.639+0000 7f4de596d640 10 monclient: monmap: 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.640+0000 7f4de596d640 1 librados: starting msgr at 2024-03-29T10:39:57.629 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.640+0000 7f4de596d640 1 librados: starting objecter 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.640+0000 7f4de596d640 1 Processor -- start 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.640+0000 7f4de596d640 1 -- start start 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.640+0000 7f4de596d640 1 librados: setting wanted keys 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.640+0000 7f4de596d640 1 librados: calling monclient init 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.640+0000 7f4de596d640 10 monclient: init 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.642+0000 7f4de596d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.642+0000 7f4de596d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:57.630 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.642+0000 7f4de596d640 10 monclient(hunting): picked mon.noname-c con 0x7f4dd8006a70 addr v1:172.21.15.94:6790/0 2024-03-29T10:39:57.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.643+0000 7f4de596d640 10 monclient(hunting): picked mon.noname-a con 0x7f4dd8029c30 addr v1:172.21.15.94:6789/0 2024-03-29T10:39:57.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de496b640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f4dd8006a70 0x7f4dd81582d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41193/0 (socket says 172.21.15.94:41193) 2024-03-29T10:39:57.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de596d640 10 monclient(hunting): picked mon.noname-b con 0x7f4dd80c1d10 addr v1:172.21.15.173:6789/0 2024-03-29T10:39:57.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de596d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd8157860 con 0x7f4dd8029c30 2024-03-29T10:39:57.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de596d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd8157b40 con 0x7f4dd80c1d10 2024-03-29T10:39:57.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de596d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd80c22c0 con 0x7f4dd8006a70 2024-03-29T10:39:57.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de596d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:57.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de596d640 10 monclient(hunting): authenticate will time out at 848.915841s 2024-03-29T10:39:57.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.647+0000 7f4de496b640 1 -- 172.21.15.94:0/2685649502 learned_addr learned my addr 172.21.15.94:0/2685649502 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:39:57.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.649+0000 7f4ddd7fa640 10 client.?.objecter ms_handle_connect 0x7f4dd8006a70 2024-03-29T10:39:57.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.649+0000 7f4ddd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:57.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.649+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401582694 0 0) 0x7f4dd80c22c0 con 0x7f4dd8006a70 2024-03-29T10:39:57.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.649+0000 7f4ddd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:57.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.649+0000 7f4ddd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:57.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.650+0000 7f4ddd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:57.634 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.650+0000 7f4ddd7fa640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:39:57.634 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.650+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc4004eb0 con 0x7f4dd8006a70 2024-03-29T10:39:57.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 client.?.objecter ms_handle_connect 0x7f4dd8029c30 2024-03-29T10:39:57.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949745668 0 0) 0x7f4dd8157860 con 0x7f4dd8029c30 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd80c22c0 con 0x7f4dd8029c30 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3601216810 0 0) 0x7f4dc4004eb0 con 0x7f4dd8006a70 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dd8157860 con 0x7f4dd8006a70 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4dd4001a10 con 0x7f4dd8006a70 2024-03-29T10:39:57.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:57.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.652+0000 7f4ddd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4140807401 0 0) 0x7f4dd80c22c0 con 0x7f4dd8029c30 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dc4004eb0 con 0x7f4dd8029c30 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4dcc001a60 con 0x7f4dd8029c30 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.638 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:57.639 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:57.639 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:57.639 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:57.639 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:57.639 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:57.639 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 786644802 0 0) 0x7f4dd8157860 con 0x7f4dd8006a70 2024-03-29T10:39:57.639 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.653+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 >> v1:172.21.15.173:6789/0 conn(0x7f4dd80c1d10 legacy=0x7f4dd8155850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 >> v1:172.21.15.94:6789/0 conn(0x7f4dd8029c30 legacy=0x7f4dd80c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 1 monclient: found mon.c 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd80c2450 con 0x7f4dd8006a70 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:39:57.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4ddd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4de596d640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4de596d640 10 monclient: _renew_subs 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.655+0000 7f4de596d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd8155f60 con 0x7f4dd8006a70 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4de596d640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dd4001c50 con 0x7f4dd8006a70 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4ddd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4dd4003a60 con 0x7f4dd8006a70 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4de596d640 10 monclient: _renew_subs 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4de596d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd8156490 con 0x7f4dd8006a70 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4dde7fc640 4 set_mon_vals no callback set 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4de596d640 1 librados: init done 2024-03-29T10:39:57.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4ddd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4ddd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.656+0000 7f4ddd7fa640 10 monclient: dump: 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.658+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f4dd4003bc0 con 0x7f4dd8006a70 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.658+0000 7f4de596d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:57.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.658+0000 7f4de596d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:57.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.658+0000 7f4de596d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:57.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.658+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd8004cd0 con 0x7f4dd8006a70 2024-03-29T10:39:57.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.659+0000 7f4ddffff640 10 monclient: get_auth_request con 0x7f4dc406dea0 auth_method 0 2024-03-29T10:39:57.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.662+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f4dd4084ad0 con 0x7f4dd8006a70 2024-03-29T10:39:57.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.662+0000 7f4ddd7fa640 10 client.4274.objecter ms_dispatch 0x7f4de0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:57.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.662+0000 7f4ddd7fa640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:57.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.662+0000 7f4ddd7fa640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:57.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.662+0000 7f4ddd7fa640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T10:39:57.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.662+0000 7f4ddd7fa640 10 client.4274.objecter ms_handle_connect 0x7f4dc406dea0 2024-03-29T10:39:57.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.669+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4dd40503b0 con 0x7f4dd8006a70 2024-03-29T10:39:57.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.669+0000 7f4ddd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:57.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:57.669+0000 7f4ddd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:57.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.001+0000 7f4de596d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:57.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.001+0000 7f4de596d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:57.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.001+0000 7f4de596d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:39:57.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.001+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dd8155470 con 0x7f4dd8006a70 2024-03-29T10:39:57.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.006+0000 7f4ddd7fa640 1 -- 172.21.15.94:0/2685649502 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f4dd403efe0 con 0x7f4dd8006a70 2024-03-29T10:39:57.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.006+0000 7f4ddd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:57.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.006+0000 7f4ddd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:57.991 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:39:57.991 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:57.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.011+0000 7f4de596d640 10 librados: watch_flush enter 2024-03-29T10:39:57.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.011+0000 7f4de596d640 10 librados: watch_flush exit 2024-03-29T10:39:57.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.011+0000 7f4de596d640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T10:39:57.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.011+0000 7f4de596d640 10 client.4274.objecter successfully canceled tick 2024-03-29T10:39:57.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 >> v1:172.21.15.94:6816/3829934808 conn(0x7f4dc406dea0 legacy=0x7f4dc4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:57.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 10 monclient: shutdown 2024-03-29T10:39:57.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:57.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 >> v1:172.21.15.94:6790/0 conn(0x7f4dd8006a70 legacy=0x7f4dd81582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:57.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 shutdown_connections 2024-03-29T10:39:57.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 >> 172.21.15.94:0/2685649502 conn(0x7f4dd80058f0 msgr2=0x7f4dd8033ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:57.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 shutdown_connections 2024-03-29T10:39:57.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 1 -- 172.21.15.94:0/2685649502 wait complete. 2024-03-29T10:39:57.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:39:58.012+0000 7f4de596d640 1 librados: shutdown 2024-03-29T10:39:58.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:04.821 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:04.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.995+0000 7f58c9d4d640 10 monclient: get_monmap_and_config 2024-03-29T10:40:04.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.995+0000 7f58c9d4d640 10 monclient: build_initial_monmap 2024-03-29T10:40:04.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.995+0000 7f58c9d4d640 10 monclient: monmap: 2024-03-29T10:40:04.979 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:04.979 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:04.979 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:04.996237+0000 2024-03-29T10:40:04.979 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:04.996237+0000 2024-03-29T10:40:04.980 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:04.980 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:04.980 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:04.980 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:04.980 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:04.980 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:04.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 1 Processor -- start 2024-03-29T10:40:04.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 1 -- start start 2024-03-29T10:40:04.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 10 monclient: init 2024-03-29T10:40:04.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:04.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:04.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f58c41457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:04.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f58c413f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:04.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f58c40a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:04.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.998+0000 7f58c9d4d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c40aaef0 con 0x7f58c40a9d70 2024-03-29T10:40:04.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.999+0000 7f58c9d4d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c4145bd0 con 0x7f58c413f650 2024-03-29T10:40:04.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.999+0000 7f58c9d4d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c4145eb0 con 0x7f58c41457f0 2024-03-29T10:40:04.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.999+0000 7f58c9d4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:04.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:04.999+0000 7f58c9d4d640 10 monclient(hunting): authenticate will time out at 856.267108s 2024-03-29T10:40:04.986 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.002+0000 7f58c37fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f58c41457f0 0x7f58c413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35879/0 (socket says 172.21.15.94:35879) 2024-03-29T10:40:04.986 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.002+0000 7f58c37fe640 1 -- 172.21.15.94:0/1229316276 learned_addr learned my addr 172.21.15.94:0/1229316276 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:04.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860051714 0 0) 0x7f58c4145eb0 con 0x7f58c41457f0 2024-03-29T10:40:04.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:04.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:04.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:04.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:40:04.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac003520 con 0x7f58c41457f0 2024-03-29T10:40:04.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362532779 0 0) 0x7f58ac003520 con 0x7f58c41457f0 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 >> v1:172.21.15.173:6789/0 conn(0x7f58c413f650 legacy=0x7f58c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 >> v1:172.21.15.94:6789/0 conn(0x7f58c40a9d70 legacy=0x7f58c414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 1 monclient: found mon.noname-b 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.005+0000 7f58c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c4146340 con 0x7f58c41457f0 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f58b4001690 con 0x7f58c41457f0 2024-03-29T10:40:04.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 10 monclient: dump: 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 1 -- 172.21.15.94:0/1229316276 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b40018b0 con 0x7f58c41457f0 2024-03-29T10:40:04.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:04.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c9d4d640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c9d4d640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.006+0000 7f58c9d4d640 4 set_mon_vals no callback set 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.007+0000 7f58c9d4d640 10 monclient: shutdown 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.007+0000 7f58c9d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.007+0000 7f58c9d4d640 1 -- 172.21.15.94:0/1229316276 >> v1:172.21.15.94:6790/0 conn(0x7f58c41457f0 legacy=0x7f58c413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.009+0000 7f58c9d4d640 1 -- 172.21.15.94:0/1229316276 shutdown_connections 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.009+0000 7f58c9d4d640 1 -- 172.21.15.94:0/1229316276 >> 172.21.15.94:0/1229316276 conn(0x7f58c4066a00 msgr2=0x7f58c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:04.993 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.009+0000 7f58c9d4d640 1 -- 172.21.15.94:0/1229316276 shutdown_connections 2024-03-29T10:40:04.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.009+0000 7f58c9d4d640 1 -- 172.21.15.94:0/1229316276 wait complete. 2024-03-29T10:40:04.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.010+0000 7f58c9d4d640 10 monclient: build_initial_monmap 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.010+0000 7f58c9d4d640 10 monclient: monmap: 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.011+0000 7f58c9d4d640 1 librados: starting msgr at 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.011+0000 7f58c9d4d640 1 librados: starting objecter 2024-03-29T10:40:04.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.011+0000 7f58c9d4d640 1 Processor -- start 2024-03-29T10:40:04.998 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.012+0000 7f58c9d4d640 1 -- start start 2024-03-29T10:40:04.998 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.012+0000 7f58c9d4d640 1 librados: setting wanted keys 2024-03-29T10:40:04.998 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.012+0000 7f58c9d4d640 1 librados: calling monclient init 2024-03-29T10:40:04.998 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.012+0000 7f58c9d4d640 10 monclient: init 2024-03-29T10:40:04.998 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.013+0000 7f58c9d4d640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:04.998 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.013+0000 7f58c9d4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:04.998 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.013+0000 7f58c9d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f58c40a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:05.003 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.014+0000 7f58c9d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f58c413f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:05.003 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.014+0000 7f58c9d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f58c41457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.014+0000 7f58c9d4d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c4140100 con 0x7f58c40a9d70 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.014+0000 7f58c9d4d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c406ed80 con 0x7f58c413f650 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.014+0000 7f58c9d4d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c406fdc0 con 0x7f58c41457f0 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.014+0000 7f58c9d4d640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.014+0000 7f58c9d4d640 10 monclient(hunting): authenticate will time out at 856.282849s 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.019+0000 7f58c3fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f58c41457f0 0x7f58c41e42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35951/0 (socket says 172.21.15.94:35951) 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.019+0000 7f58c3fff640 1 -- 172.21.15.94:0/586148072 learned_addr learned my addr 172.21.15.94:0/586148072 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.019+0000 7f58abfff640 10 client.?.objecter ms_handle_connect 0x7f58c40a9d70 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.019+0000 7f58abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.019+0000 7f58abfff640 10 client.?.objecter ms_handle_connect 0x7f58c41457f0 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.019+0000 7f58abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:05.004 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.019+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340471826 0 0) 0x7f58c4140100 con 0x7f58c40a9d70 2024-03-29T10:40:05.006 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.020+0000 7f58abfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.020+0000 7f58abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.020+0000 7f58abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.020+0000 7f58abfff640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b0004eb0 con 0x7f58c40a9d70 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 10 client.?.objecter ms_handle_connect 0x7f58c413f650 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824575286 0 0) 0x7f58c406fdc0 con 0x7f58c41457f0 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c4140100 con 0x7f58c41457f0 2024-03-29T10:40:05.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.021+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884505035 0 0) 0x7f58c4140100 con 0x7f58c41457f0 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.022+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58c406fdc0 con 0x7f58c41457f0 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.022+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f58b80018f0 con 0x7f58c41457f0 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.022+0000 7f58abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.022+0000 7f58abfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.022+0000 7f58abfff640 10 monclient(hunting): dump: 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:05.008 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:05.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:05.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1688360585 0 0) 0x7f58c406fdc0 con 0x7f58c41457f0 2024-03-29T10:40:05.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 >> v1:172.21.15.173:6789/0 conn(0x7f58c413f650 legacy=0x7f58c4071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 >> v1:172.21.15.94:6789/0 conn(0x7f58c40a9d70 legacy=0x7f58c406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 1 monclient: found mon.c 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c406ea10 con 0x7f58c41457f0 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: _finish_auth 0 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:05.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 5 monclient: authenticate success, global_id 4280 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 10 monclient: _renew_subs 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c4142d60 con 0x7f58c41457f0 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 10 monclient: _renew_subs 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c406e780 con 0x7f58c41457f0 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c9d4d640 1 librados: init done 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b8003610 con 0x7f58c41457f0 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f58b80039a0 con 0x7f58c41457f0 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:05.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58abfff640 10 monclient: dump: 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.023+0000 7f58c1ffb640 4 set_mon_vals no callback set 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.024+0000 7f58c9d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.025+0000 7f58c9d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:05.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.025+0000 7f58c9d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:05.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.025+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a4005080 con 0x7f58c41457f0 2024-03-29T10:40:05.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.025+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f58b8003b80 con 0x7f58c41457f0 2024-03-29T10:40:05.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.026+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f58b8084a10 con 0x7f58c41457f0 2024-03-29T10:40:05.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.026+0000 7f58c37fe640 10 monclient: get_auth_request con 0x7f58b006dc40 auth_method 0 2024-03-29T10:40:05.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.027+0000 7f58abfff640 10 client.4280.objecter ms_dispatch 0x7f58c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:05.014 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.027+0000 7f58abfff640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:05.014 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.027+0000 7f58abfff640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:05.014 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.027+0000 7f58abfff640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T10:40:05.014 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.027+0000 7f58abfff640 10 client.4280.objecter ms_handle_connect 0x7f58b006dc40 2024-03-29T10:40:05.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.034+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58b80502f0 con 0x7f58c41457f0 2024-03-29T10:40:05.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.034+0000 7f58abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:05.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.034+0000 7f58abfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:05.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.354+0000 7f58c9d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:05.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.354+0000 7f58c9d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:05.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.354+0000 7f58c9d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:05.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.354+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58a4005330 con 0x7f58c41457f0 2024-03-29T10:40:05.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.355+0000 7f58abfff640 1 -- 172.21.15.94:0/586148072 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f58b8052220 con 0x7f58c41457f0 2024-03-29T10:40:05.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.355+0000 7f58abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:05.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.355+0000 7f58abfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:05.341 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:05.341 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:05.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.358+0000 7f58c9d4d640 10 librados: watch_flush enter 2024-03-29T10:40:05.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.359+0000 7f58c9d4d640 10 librados: watch_flush exit 2024-03-29T10:40:05.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.359+0000 7f58c9d4d640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T10:40:05.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.359+0000 7f58c9d4d640 10 client.4280.objecter successfully canceled tick 2024-03-29T10:40:05.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.359+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 >> v1:172.21.15.94:6816/3829934808 conn(0x7f58b006dc40 legacy=0x7f58b0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:05.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.359+0000 7f58c9d4d640 10 monclient: shutdown 2024-03-29T10:40:05.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.359+0000 7f58c9d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:05.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.359+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 >> v1:172.21.15.94:6790/0 conn(0x7f58c41457f0 legacy=0x7f58c41e42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:05.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.360+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 shutdown_connections 2024-03-29T10:40:05.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.360+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 >> 172.21.15.94:0/586148072 conn(0x7f58c4066a00 msgr2=0x7f58c4139020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:05.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.361+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 shutdown_connections 2024-03-29T10:40:05.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.361+0000 7f58c9d4d640 1 -- 172.21.15.94:0/586148072 wait complete. 2024-03-29T10:40:05.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:05.361+0000 7f58c9d4d640 1 librados: shutdown 2024-03-29T10:40:05.363 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:12.170 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.348+0000 7f0c89aba640 10 monclient: get_monmap_and_config 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.348+0000 7f0c89aba640 10 monclient: build_initial_monmap 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.348+0000 7f0c89aba640 10 monclient: monmap: 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:12.349220+0000 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:12.349220+0000 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:12.332 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:12.333 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:12.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.349+0000 7f0c89aba640 1 Processor -- start 2024-03-29T10:40:12.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.349+0000 7f0c89aba640 1 -- start start 2024-03-29T10:40:12.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.349+0000 7f0c89aba640 10 monclient: init 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 10 monclient(hunting): picked mon.noname-c con 0x7f0c8406ade0 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 10 monclient(hunting): picked mon.noname-b con 0x7f0c840afa90 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 10 monclient(hunting): picked mon.noname-a con 0x7f0c840ae550 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840b0b20 con 0x7f0c840ae550 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8413d8c0 con 0x7f0c8406ade0 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8413da50 con 0x7f0c840afa90 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c89aba640 10 monclient(hunting): authenticate will time out at 863.619388s 2024-03-29T10:40:12.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c837fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0c8406ade0 0x7f0c84143390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:55484/0 (socket says 172.21.15.94:55484) 2024-03-29T10:40:12.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.351+0000 7f0c837fe640 1 -- 172.21.15.94:0/2805182781 learned_addr learned my addr 172.21.15.94:0/2805182781 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:12.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808944487 0 0) 0x7f0c8413d8c0 con 0x7f0c8406ade0 2024-03-29T10:40:12.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:12.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:12.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c003520 con 0x7f0c8406ade0 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763777137 0 0) 0x7f0c840b0b20 con 0x7f0c840ae550 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c8413d8c0 con 0x7f0c840ae550 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443463056 0 0) 0x7f0c8413da50 con 0x7f0c840afa90 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c840b0b20 con 0x7f0c840afa90 2024-03-29T10:40:12.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1349922015 0 0) 0x7f0c6c003520 con 0x7f0c8406ade0 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 >> v1:172.21.15.94:6790/0 conn(0x7f0c840afa90 legacy=0x7f0c8413d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 >> v1:172.21.15.94:6789/0 conn(0x7f0c840ae550 legacy=0x7f0c8414b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 monclient: found mon.noname-c 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84072570 con 0x7f0c8406ade0 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c89aba640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c89aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0c7c001a10 con 0x7f0c8406ade0 2024-03-29T10:40:12.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c827fc640 10 monclient: dump: 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:12.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.352+0000 7f0c89aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c7c001c30 con 0x7f0c8406ade0 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c89aba640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c89aba640 4 set_mon_vals no callback set 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c827fc640 1 -- 172.21.15.94:0/2805182781 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0c7c001fc0 con 0x7f0c8406ade0 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.353+0000 7f0c827fc640 10 monclient: dump: 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:12.341 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.354+0000 7f0c89aba640 10 monclient: shutdown 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.354+0000 7f0c89aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.354+0000 7f0c89aba640 1 -- 172.21.15.94:0/2805182781 >> v1:172.21.15.173:6789/0 conn(0x7f0c8406ade0 legacy=0x7f0c84143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.354+0000 7f0c89aba640 1 -- 172.21.15.94:0/2805182781 shutdown_connections 2024-03-29T10:40:12.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.354+0000 7f0c89aba640 1 -- 172.21.15.94:0/2805182781 >> 172.21.15.94:0/2805182781 conn(0x7f0c840b4930 msgr2=0x7f0c840b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.354+0000 7f0c89aba640 1 -- 172.21.15.94:0/2805182781 shutdown_connections 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.354+0000 7f0c89aba640 1 -- 172.21.15.94:0/2805182781 wait complete. 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.355+0000 7f0c89aba640 10 monclient: build_initial_monmap 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.355+0000 7f0c89aba640 10 monclient: monmap: 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:12.343 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.356+0000 7f0c89aba640 1 librados: starting msgr at 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.356+0000 7f0c89aba640 1 librados: starting objecter 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.356+0000 7f0c89aba640 1 Processor -- start 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.356+0000 7f0c89aba640 1 -- start start 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.356+0000 7f0c89aba640 1 librados: setting wanted keys 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.356+0000 7f0c89aba640 1 librados: calling monclient init 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.356+0000 7f0c89aba640 10 monclient: init 2024-03-29T10:40:12.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.357+0000 7f0c89aba640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.357+0000 7f0c89aba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.357+0000 7f0c89aba640 10 monclient(hunting): picked mon.noname-c con 0x7f0c8406ade0 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.357+0000 7f0c89aba640 10 monclient(hunting): picked mon.noname-a con 0x7f0c840ae550 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.357+0000 7f0c89aba640 10 monclient(hunting): picked mon.noname-b con 0x7f0c840afa90 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.357+0000 7f0c89aba640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c841364e0 con 0x7f0c840ae550 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c89aba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c84143d80 con 0x7f0c840afa90 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c89aba640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c841432d0 con 0x7f0c8406ade0 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c89aba640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:12.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c89aba640 10 monclient(hunting): authenticate will time out at 863.626131s 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c837fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f0c8406ade0 0x7f0c84142860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:33080/0 (socket says 172.21.15.94:33080) 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c837fe640 1 -- 172.21.15.94:0/3692350104 learned_addr learned my addr 172.21.15.94:0/3692350104 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 client.?.objecter ms_handle_connect 0x7f0c8406ade0 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 client.?.objecter ms_handle_connect 0x7f0c840ae550 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 client.?.objecter ms_handle_connect 0x7f0c840afa90 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141665155 0 0) 0x7f0c84143d80 con 0x7f0c840afa90 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:40:12.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.358+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c70004eb0 con 0x7f0c840afa90 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607426694 0 0) 0x7f0c841364e0 con 0x7f0c840ae550 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c84143d80 con 0x7f0c840ae550 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3408340203 0 0) 0x7f0c70004eb0 con 0x7f0c840afa90 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c841364e0 con 0x7f0c840afa90 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100333912 0 0) 0x7f0c841432d0 con 0x7f0c8406ade0 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:40:12.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c70004eb0 con 0x7f0c8406ade0 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0c740018b0 con 0x7f0c840afa90 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): dump: 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:12.350 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 419583655 0 0) 0x7f0c84143d80 con 0x7f0c840ae550 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.359+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c841432d0 con 0x7f0c840ae550 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0c78001810 con 0x7f0c840ae550 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 10 monclient(hunting): dump: 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:12.351 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4158734323 0 0) 0x7f0c70004eb0 con 0x7f0c8406ade0 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c84143d80 con 0x7f0c8406ade0 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2816996956 0 0) 0x7f0c841364e0 con 0x7f0c840afa90 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 >> v1:172.21.15.94:6790/0 conn(0x7f0c8406ade0 legacy=0x7f0c84142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 >> v1:172.21.15.94:6789/0 conn(0x7f0c840ae550 legacy=0x7f0c841402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 monclient: found mon.b 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:12.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84142f70 con 0x7f0c840afa90 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 10 monclient: _finish_auth 0 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 10 monclient: _renew_subs 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c84143ff0 con 0x7f0c840afa90 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 10 monclient: _renew_subs 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c841373d0 con 0x7f0c840afa90 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.360+0000 7f0c89aba640 1 librados: init done 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.361+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c74003610 con 0x7f0c840afa90 2024-03-29T10:40:12.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.361+0000 7f0c6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.361+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0c740039e0 con 0x7f0c840afa90 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.361+0000 7f0c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.361+0000 7f0c6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.361+0000 7f0c6bfff640 10 monclient: dump: 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:12.354 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:12.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.361+0000 7f0c817fa640 4 set_mon_vals no callback set 2024-03-29T10:40:12.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.362+0000 7f0c89aba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:12.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.362+0000 7f0c89aba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:12.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.362+0000 7f0c89aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:12.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.362+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c64005080 con 0x7f0c840afa90 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.364+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f0c74003b40 con 0x7f0c840afa90 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.365+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0c740836d0 con 0x7f0c840afa90 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.365+0000 7f0c6bfff640 10 client.4270.objecter ms_dispatch 0x7f0c84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.365+0000 7f0c6bfff640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.365+0000 7f0c6bfff640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.365+0000 7f0c837fe640 10 monclient: get_auth_request con 0x7f0c7006e0a0 auth_method 0 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.365+0000 7f0c6bfff640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.366+0000 7f0c6bfff640 10 client.4270.objecter ms_handle_connect 0x7f0c7006e0a0 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.370+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c7404efb0 con 0x7f0c840afa90 2024-03-29T10:40:12.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.370+0000 7f0c6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:12.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.370+0000 7f0c6bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:12.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.795+0000 7f0c89aba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:12.780 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.795+0000 7f0c89aba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:12.780 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.795+0000 7f0c89aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:12.780 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.795+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c64005330 con 0x7f0c840afa90 2024-03-29T10:40:12.780 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:12.780 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:12.781 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.796+0000 7f0c6bfff640 1 -- 172.21.15.94:0/3692350104 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f0c74050ee0 con 0x7f0c840afa90 2024-03-29T10:40:12.781 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.796+0000 7f0c6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:12.781 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.796+0000 7f0c6bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:12.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 10 librados: watch_flush enter 2024-03-29T10:40:12.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 10 librados: watch_flush exit 2024-03-29T10:40:12.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:40:12.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:40:12.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 >> v1:172.21.15.94:6816/3829934808 conn(0x7f0c7006e0a0 legacy=0x7f0c70070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:12.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 10 monclient: shutdown 2024-03-29T10:40:12.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:12.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.800+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 >> v1:172.21.15.173:6789/0 conn(0x7f0c840afa90 legacy=0x7f0c841e6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:12.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.801+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 shutdown_connections 2024-03-29T10:40:12.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.801+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 >> 172.21.15.94:0/3692350104 conn(0x7f0c840b4930 msgr2=0x7f0c841395f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:12.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.801+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 shutdown_connections 2024-03-29T10:40:12.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.801+0000 7f0c89aba640 1 -- 172.21.15.94:0/3692350104 wait complete. 2024-03-29T10:40:12.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:12.802+0000 7f0c89aba640 1 librados: shutdown 2024-03-29T10:40:12.802 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:19.609 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.760+0000 7f386a315640 10 monclient: get_monmap_and_config 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.760+0000 7f386a315640 10 monclient: build_initial_monmap 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.760+0000 7f386a315640 10 monclient: monmap: 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:19.761295+0000 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:19.761295+0000 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:19.744 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:19.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.761+0000 7f386a315640 1 Processor -- start 2024-03-29T10:40:19.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.761+0000 7f386a315640 1 -- start start 2024-03-29T10:40:19.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.761+0000 7f386a315640 10 monclient: init 2024-03-29T10:40:19.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 10 monclient(hunting): picked mon.noname-c con 0x7f38641457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 10 monclient(hunting): picked mon.noname-a con 0x7f386413f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 10 monclient(hunting): picked mon.noname-b con 0x7f38640a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38640aaef0 con 0x7f386413f650 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3864145bd0 con 0x7f38641457f0 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3864145eb0 con 0x7f38640a9d70 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f386a315640 10 monclient(hunting): authenticate will time out at 871.030555s 2024-03-29T10:40:19.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.762+0000 7f3863fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f38641457f0 0x7f386413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:40796/0 (socket says 172.21.15.94:40796) 2024-03-29T10:40:19.747 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.763+0000 7f3863fff640 1 -- 172.21.15.94:0/474370410 learned_addr learned my addr 172.21.15.94:0/474370410 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:19.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.764+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522311834 0 0) 0x7f3864145bd0 con 0x7f38641457f0 2024-03-29T10:40:19.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.764+0000 7f3862ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:19.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.764+0000 7f3862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:19.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.764+0000 7f3862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:19.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.764+0000 7f3862ffd640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:40:19.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f384c003520 con 0x7f38641457f0 2024-03-29T10:40:19.749 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960520649 0 0) 0x7f384c003520 con 0x7f38641457f0 2024-03-29T10:40:19.749 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 >> v1:172.21.15.94:6790/0 conn(0x7f38640a9d70 legacy=0x7f386414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:19.749 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 >> v1:172.21.15.94:6789/0 conn(0x7f386413f650 legacy=0x7f38640b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:19.749 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:19.749 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 1 monclient: found mon.noname-c 2024-03-29T10:40:19.749 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:19.749 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3864146340 con 0x7f38641457f0 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f386a315640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f386a315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3858001690 con 0x7f38641457f0 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f3862ffd640 10 monclient: dump: 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.750 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.751 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:19.751 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:19.751 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:19.751 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:19.751 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:19.751 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:19.751 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.765+0000 7f386a315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:19.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3858001920 con 0x7f38641457f0 2024-03-29T10:40:19.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f3862ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:19.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f386a315640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:19.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f386a315640 4 set_mon_vals no callback set 2024-03-29T10:40:19.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f3862ffd640 1 -- 172.21.15.94:0/474370410 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3858001cb0 con 0x7f38641457f0 2024-03-29T10:40:19.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f3862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:19.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f3862ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f3862ffd640 10 monclient: dump: 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f386a315640 10 monclient: shutdown 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f386a315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.767+0000 7f386a315640 1 -- 172.21.15.94:0/474370410 >> v1:172.21.15.173:6789/0 conn(0x7f38641457f0 legacy=0x7f386413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:19.753 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.768+0000 7f386a315640 1 -- 172.21.15.94:0/474370410 shutdown_connections 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.768+0000 7f386a315640 1 -- 172.21.15.94:0/474370410 >> 172.21.15.94:0/474370410 conn(0x7f3864066a00 msgr2=0x7f3864066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.768+0000 7f386a315640 1 -- 172.21.15.94:0/474370410 shutdown_connections 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.768+0000 7f386a315640 1 -- 172.21.15.94:0/474370410 wait complete. 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.768+0000 7f386a315640 10 monclient: build_initial_monmap 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.768+0000 7f386a315640 10 monclient: monmap: 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:19.754 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:19.755 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:19.756 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.769+0000 7f386a315640 1 librados: starting msgr at 2024-03-29T10:40:19.756 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.769+0000 7f386a315640 1 librados: starting objecter 2024-03-29T10:40:19.756 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.769+0000 7f386a315640 1 Processor -- start 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.769+0000 7f386a315640 1 -- start start 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.769+0000 7f386a315640 1 librados: setting wanted keys 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.769+0000 7f386a315640 1 librados: calling monclient init 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.769+0000 7f386a315640 10 monclient: init 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 10 monclient(hunting): picked mon.noname-c con 0x7f38640a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 10 monclient(hunting): picked mon.noname-a con 0x7f386413f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 10 monclient(hunting): picked mon.noname-b con 0x7f38641457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38641427d0 con 0x7f386413f650 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38640b14e0 con 0x7f38641457f0 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38641422b0 con 0x7f38640a9d70 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:19.757 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f386a315640 10 monclient(hunting): authenticate will time out at 871.038721s 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f3863fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f38640a9d70 0x7f38640b0860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:52882/0 (socket says 172.21.15.94:52882) 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f3863fff640 1 -- 172.21.15.94:0/4263481846 learned_addr learned my addr 172.21.15.94:0/4263481846 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f3860ff9640 10 client.?.objecter ms_handle_connect 0x7f38640a9d70 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f3860ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.770+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400768985 0 0) 0x7f38641422b0 con 0x7f38640a9d70 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3850004eb0 con 0x7f38640a9d70 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 client.?.objecter ms_handle_connect 0x7f38641457f0 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4076987536 0 0) 0x7f3850004eb0 con 0x7f38640a9d70 2024-03-29T10:40:19.758 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38641422b0 con 0x7f38640a9d70 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3858001650 con 0x7f38640a9d70 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): dump: 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:19.759 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403894657 0 0) 0x7f38640b14e0 con 0x7f38641457f0 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3850004eb0 con 0x7f38641457f0 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 903699912 0 0) 0x7f38641422b0 con 0x7f38640a9d70 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 >> v1:172.21.15.173:6789/0 conn(0x7f38641457f0 legacy=0x7f38641e27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.771+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 >> v1:172.21.15.94:6789/0 conn(0x7f386413f650 legacy=0x7f38641d6f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.772+0000 7f3860ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:19.760 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.772+0000 7f3860ff9640 1 monclient: found mon.c 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f3860ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f3860ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38641417d0 con 0x7f38640a9d70 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f3860ff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f3860ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f3860ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f386a315640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:40:19.762 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f386a315640 10 monclient: _renew_subs 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f386a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.773+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3864141a20 con 0x7f38640a9d70 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f386a315640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f386a315640 10 monclient: _renew_subs 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f386a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38640b1150 con 0x7f38640a9d70 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f386a315640 1 librados: init done 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3858004a90 con 0x7f38640a9d70 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f3860ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f38627fc640 4 set_mon_vals no callback set 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3858004ee0 con 0x7f38640a9d70 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f3860ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f3860ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:19.763 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.774+0000 7f3860ff9640 10 monclient: dump: 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.775+0000 7f386a315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.775+0000 7f386a315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.775+0000 7f386a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.775+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3844005080 con 0x7f38640a9d70 2024-03-29T10:40:19.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.776+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f3858005040 con 0x7f38640a9d70 2024-03-29T10:40:19.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.776+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f3858084bd0 con 0x7f38640a9d70 2024-03-29T10:40:19.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.776+0000 7f3860ff9640 10 client.4289.objecter ms_dispatch 0x7f3864000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:19.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.776+0000 7f3860ff9640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:19.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.776+0000 7f3860ff9640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:19.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.776+0000 7f38637fe640 10 monclient: get_auth_request con 0x7f385006dd60 auth_method 0 2024-03-29T10:40:19.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.776+0000 7f3860ff9640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T10:40:19.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.777+0000 7f3860ff9640 10 client.4289.objecter ms_handle_connect 0x7f385006dd60 2024-03-29T10:40:19.766 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.783+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38580504b0 con 0x7f38640a9d70 2024-03-29T10:40:19.766 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.783+0000 7f3860ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:19.767 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:19.783+0000 7f3860ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:20.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.096+0000 7f386a315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:20.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.096+0000 7f386a315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:20.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.096+0000 7f386a315640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:20.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.096+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3844005330 con 0x7f38640a9d70 2024-03-29T10:40:20.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.098+0000 7f3860ff9640 1 -- 172.21.15.94:0/4263481846 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f38580523e0 con 0x7f38640a9d70 2024-03-29T10:40:20.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.098+0000 7f3860ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:20.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.098+0000 7f3860ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:20.082 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:20.083 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 10 librados: watch_flush enter 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 10 librados: watch_flush exit 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 10 client.4289.objecter successfully canceled tick 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 >> v1:172.21.15.94:6816/3829934808 conn(0x7f385006dd60 legacy=0x7f3850070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 10 monclient: shutdown 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:20.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.101+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 >> v1:172.21.15.94:6790/0 conn(0x7f38640a9d70 legacy=0x7f38640b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:20.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.102+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 shutdown_connections 2024-03-29T10:40:20.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.102+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 >> 172.21.15.94:0/4263481846 conn(0x7f3864066a00 msgr2=0x7f38640b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:20.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.102+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 shutdown_connections 2024-03-29T10:40:20.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.102+0000 7f386a315640 1 -- 172.21.15.94:0/4263481846 wait complete. 2024-03-29T10:40:20.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:20.102+0000 7f386a315640 1 librados: shutdown 2024-03-29T10:40:20.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:26.906 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:27.038 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.054+0000 7f88fd0fb640 10 monclient: get_monmap_and_config 2024-03-29T10:40:27.038 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.054+0000 7f88fd0fb640 10 monclient: build_initial_monmap 2024-03-29T10:40:27.038 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.054+0000 7f88fd0fb640 10 monclient: monmap: 2024-03-29T10:40:27.038 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:27.038 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:27.038 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:27.055572+0000 2024-03-29T10:40:27.039 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:27.055572+0000 2024-03-29T10:40:27.039 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:27.039 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:27.039 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:27.039 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:27.039 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:27.039 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:27.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.055+0000 7f88fd0fb640 1 Processor -- start 2024-03-29T10:40:27.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.055+0000 7f88fd0fb640 1 -- start start 2024-03-29T10:40:27.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.055+0000 7f88fd0fb640 10 monclient: init 2024-03-29T10:40:27.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f88f81422b0 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f88f8142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f88f80b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f8142c80 con 0x7f88f8142690 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f80aae40 con 0x7f88f81422b0 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f813fbd0 con 0x7f88f80b17d0 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.056+0000 7f88fd0fb640 10 monclient(hunting): authenticate will time out at 878.324882s 2024-03-29T10:40:27.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f759e640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f88f80b17d0 0x7f88f80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:37784/0 (socket says 172.21.15.94:37784) 2024-03-29T10:40:27.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f759e640 1 -- 172.21.15.94:0/3366711923 learned_addr learned my addr 172.21.15.94:0/3366711923 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571035688 0 0) 0x7f88f813fbd0 con 0x7f88f80b17d0 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f5d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f5d9b640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.057+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e0003520 con 0x7f88f80b17d0 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.058+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4000929122 0 0) 0x7f88e0003520 con 0x7f88f80b17d0 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.058+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 >> v1:172.21.15.173:6789/0 conn(0x7f88f81422b0 legacy=0x7f88f813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:27.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.058+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 >> v1:172.21.15.94:6789/0 conn(0x7f88f8142690 legacy=0x7f88f80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 1 monclient: found mon.noname-b 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8151940 con 0x7f88f80b17d0 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 10 monclient: _finish_auth 0 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:27.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f88f0001a10 con 0x7f88f80b17d0 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88f5d9b640 10 monclient: dump: 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:27.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88fd0fb640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.060+0000 7f88fd0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88f0001c70 con 0x7f88f80b17d0 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88f5d9b640 1 -- 172.21.15.94:0/3366711923 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f88f0002040 con 0x7f88f80b17d0 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88fd0fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88fd0fb640 4 set_mon_vals no callback set 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88f5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.061+0000 7f88f5d9b640 10 monclient: dump: 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:27.049 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.062+0000 7f88fd0fb640 10 monclient: shutdown 2024-03-29T10:40:27.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.062+0000 7f88fd0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.062+0000 7f88fd0fb640 1 -- 172.21.15.94:0/3366711923 >> v1:172.21.15.94:6790/0 conn(0x7f88f80b17d0 legacy=0x7f88f80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.062+0000 7f88fd0fb640 1 -- 172.21.15.94:0/3366711923 shutdown_connections 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.062+0000 7f88fd0fb640 1 -- 172.21.15.94:0/3366711923 >> 172.21.15.94:0/3366711923 conn(0x7f88f8066a00 msgr2=0x7f88f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.062+0000 7f88fd0fb640 1 -- 172.21.15.94:0/3366711923 shutdown_connections 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.062+0000 7f88fd0fb640 1 -- 172.21.15.94:0/3366711923 wait complete. 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.063+0000 7f88fd0fb640 10 monclient: build_initial_monmap 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.063+0000 7f88fd0fb640 10 monclient: monmap: 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:27.051 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:27.052 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:27.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.064+0000 7f88fd0fb640 1 librados: starting msgr at 2024-03-29T10:40:27.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.064+0000 7f88fd0fb640 1 librados: starting objecter 2024-03-29T10:40:27.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.064+0000 7f88fd0fb640 1 Processor -- start 2024-03-29T10:40:27.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 1 -- start start 2024-03-29T10:40:27.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 1 librados: setting wanted keys 2024-03-29T10:40:27.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 1 librados: calling monclient init 2024-03-29T10:40:27.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 10 monclient: init 2024-03-29T10:40:27.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:27.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:27.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f88f80b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:27.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f88f81422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:27.060 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.065+0000 7f88fd0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f88f8142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:27.060 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.066+0000 7f88fd0fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f80ab540 con 0x7f88f81422b0 2024-03-29T10:40:27.060 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.066+0000 7f88fd0fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f80acba0 con 0x7f88f80b17d0 2024-03-29T10:40:27.060 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.066+0000 7f88fd0fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f80ace80 con 0x7f88f8142690 2024-03-29T10:40:27.060 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.066+0000 7f88fd0fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:27.060 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.066+0000 7f88fd0fb640 10 monclient(hunting): authenticate will time out at 878.334131s 2024-03-29T10:40:27.060 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.066+0000 7f88f759e640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f88f8142690 0x7f88f8130290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:45146/0 (socket says 172.21.15.94:45146) 2024-03-29T10:40:27.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.066+0000 7f88f759e640 1 -- 172.21.15.94:0/533118904 learned_addr learned my addr 172.21.15.94:0/533118904 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:27.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.067+0000 7f88df7fe640 10 client.?.objecter ms_handle_connect 0x7f88f8142690 2024-03-29T10:40:27.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.067+0000 7f88df7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:27.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 client.?.objecter ms_handle_connect 0x7f88f81422b0 2024-03-29T10:40:27.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:27.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 client.?.objecter ms_handle_connect 0x7f88f80b17d0 2024-03-29T10:40:27.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:27.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499782735 0 0) 0x7f88f80ace80 con 0x7f88f8142690 2024-03-29T10:40:27.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:27.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:27.063 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:27.064 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:40:27.064 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e4004eb0 con 0x7f88f8142690 2024-03-29T10:40:27.064 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194624468 0 0) 0x7f88f80acba0 con 0x7f88f80b17d0 2024-03-29T10:40:27.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:27.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:27.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:27.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:40:27.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f80ace80 con 0x7f88f80b17d0 2024-03-29T10:40:27.065 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739514573 0 0) 0x7f88f80ab540 con 0x7f88f81422b0 2024-03-29T10:40:27.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:27.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:27.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:27.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:40:27.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f80acba0 con 0x7f88f81422b0 2024-03-29T10:40:27.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 26475042 0 0) 0x7f88e4004eb0 con 0x7f88f8142690 2024-03-29T10:40:27.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88f659c640 1 -- 172.21.15.94:0/533118904 >> v1:172.21.15.94:6789/0 conn(0x7f88f81422b0 legacy=0x7f88f812fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:40:27.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88f659c640 1 -- 172.21.15.94:0/533118904 >> v1:172.21.15.94:6789/0 conn(0x7f88f81422b0 legacy=0x7f88f812fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:40:27.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88f659c640 1 --1- 172.21.15.94:0/533118904 >> v1:172.21.15.94:6789/0 conn(0x7f88f81422b0 0x7f88f812fb80 :-1 s=OPENED pgs=159355 cs=1 l=1).handle_message read tag failed 2024-03-29T10:40:27.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88f659c640 1 --1- 172.21.15.94:0/533118904 >> v1:172.21.15.94:6789/0 conn(0x7f88f81422b0 0x7f88f812fb80 :-1 s=OPENED pgs=159355 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:40:27.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88f80ab540 con 0x7f88f8142690 2024-03-29T10:40:27.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6789/0 2024-03-29T10:40:27.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f88f0001f80 con 0x7f88f8142690 2024-03-29T10:40:27.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:27.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:27.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:27.068 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:27.068 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:27.068 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.068 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.069 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:27.069 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:27.069 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:27.069 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:27.069 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:27.069 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:27.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:27.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3037048423 0 0) 0x7f88f80ab540 con 0x7f88f8142690 2024-03-29T10:40:27.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 >> v1:172.21.15.173:6789/0 conn(0x7f88f80b17d0 legacy=0x7f88f81323b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:27.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 >> v1:172.21.15.94:6789/0 conn(0x7f88f81422b0 legacy=0x7f88f812fb80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:40:27.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:27.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 monclient: found mon.c 2024-03-29T10:40:27.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:27.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:27.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f80ad160 con 0x7f88f8142690 2024-03-29T10:40:27.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:27.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:27.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.069+0000 7f88df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:27.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 5 monclient: authenticate success, global_id 4295 2024-03-29T10:40:27.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 10 monclient: _renew_subs 2024-03-29T10:40:27.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:27.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f80ab790 con 0x7f88f8142690 2024-03-29T10:40:27.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:27.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 10 monclient: _renew_subs 2024-03-29T10:40:27.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:27.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f81e2e70 con 0x7f88f8142690 2024-03-29T10:40:27.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88f00020e0 con 0x7f88f8142690 2024-03-29T10:40:27.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88fd0fb640 1 librados: init done 2024-03-29T10:40:27.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:27.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f88f0005110 con 0x7f88f8142690 2024-03-29T10:40:27.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:27.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88df7fe640 10 monclient: dump: 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.070+0000 7f88f559a640 4 set_mon_vals no callback set 2024-03-29T10:40:27.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.072+0000 7f88fd0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.072+0000 7f88fd0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.072+0000 7f88fd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.073+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d8005080 con 0x7f88f8142690 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.073+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f88f001b0e0 con 0x7f88f8142690 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.074+0000 7f88f6d9d640 10 monclient: get_auth_request con 0x7f88e406de00 auth_method 0 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.074+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f88f0084f20 con 0x7f88f8142690 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.074+0000 7f88df7fe640 10 client.4295.objecter ms_dispatch 0x7f88f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.074+0000 7f88df7fe640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.074+0000 7f88df7fe640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.074+0000 7f88df7fe640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T10:40:27.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.074+0000 7f88df7fe640 10 client.4295.objecter ms_handle_connect 0x7f88e406de00 2024-03-29T10:40:27.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.080+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88f0050800 con 0x7f88f8142690 2024-03-29T10:40:27.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.080+0000 7f88df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:27.076 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.080+0000 7f88df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:27.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.390+0000 7f88fd0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:27.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.390+0000 7f88fd0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:27.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.390+0000 7f88fd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:27.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.390+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88d8005330 con 0x7f88f8142690 2024-03-29T10:40:27.375 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.391+0000 7f88df7fe640 1 -- 172.21.15.94:0/533118904 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f88f0052730 con 0x7f88f8142690 2024-03-29T10:40:27.375 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.391+0000 7f88df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:27.375 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.391+0000 7f88df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:27.375 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:27.376 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 10 librados: watch_flush enter 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 10 librados: watch_flush exit 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 10 client.4295.objecter successfully canceled tick 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 >> v1:172.21.15.94:6816/3829934808 conn(0x7f88e406de00 legacy=0x7f88e40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 10 monclient: shutdown 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 >> v1:172.21.15.94:6790/0 conn(0x7f88f8142690 legacy=0x7f88f8130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:27.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 shutdown_connections 2024-03-29T10:40:27.380 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 >> 172.21.15.94:0/533118904 conn(0x7f88f8066a00 msgr2=0x7f88f80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:27.380 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 shutdown_connections 2024-03-29T10:40:27.380 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 1 -- 172.21.15.94:0/533118904 wait complete. 2024-03-29T10:40:27.380 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:27.395+0000 7f88fd0fb640 1 librados: shutdown 2024-03-29T10:40:27.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:34.201 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:34.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.366+0000 7f5e29e99640 10 monclient: get_monmap_and_config 2024-03-29T10:40:34.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.366+0000 7f5e29e99640 10 monclient: build_initial_monmap 2024-03-29T10:40:34.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.366+0000 7f5e29e99640 10 monclient: monmap: 2024-03-29T10:40:34.353 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:34.353 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:34.353 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:34.368428+0000 2024-03-29T10:40:34.353 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:34.368428+0000 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.368+0000 7f5e29e99640 1 Processor -- start 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.368+0000 7f5e29e99640 1 -- start start 2024-03-29T10:40:34.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.368+0000 7f5e29e99640 10 monclient: init 2024-03-29T10:40:34.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:34.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:34.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 10 monclient(hunting): picked mon.noname-a con 0x7f5e1c034850 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:34.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 10 monclient(hunting): picked mon.noname-c con 0x7f5e1c0c9550 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:34.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 10 monclient(hunting): picked mon.noname-b con 0x7f5e1c02aac0 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:34.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c02c440 con 0x7f5e1c034850 2024-03-29T10:40:34.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c02aed0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c02b190 con 0x7f5e1c02aac0 2024-03-29T10:40:34.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:34.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.369+0000 7f5e29e99640 10 monclient(hunting): authenticate will time out at 885.638689s 2024-03-29T10:40:34.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.376+0000 7f5e23fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f5e1c0c9550 0x7f5e1c0cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:43327/0 (socket says 172.21.15.94:43327) 2024-03-29T10:40:34.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.376+0000 7f5e23fff640 1 -- 172.21.15.94:0/4243527664 learned_addr learned my addr 172.21.15.94:0/4243527664 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.376+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152364981 0 0) 0x7f5e1c02c440 con 0x7f5e1c034850 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e14003520 con 0x7f5e1c034850 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463345649 0 0) 0x7f5e1c02aed0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c02c440 con 0x7f5e1c0c9550 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2555610582 0 0) 0x7f5e14003520 con 0x7f5e1c034850 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 >> v1:172.21.15.94:6790/0 conn(0x7f5e1c02aac0 legacy=0x7f5e1c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 >> v1:172.21.15.173:6789/0 conn(0x7f5e1c0c9550 legacy=0x7f5e1c0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:34.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 monclient: found mon.noname-a 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c033060 con 0x7f5e1c034850 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e29e99640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:40:34.364 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e29e99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5e10001690 con 0x7f5e1c034850 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: dump: 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:34.365 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e29e99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e100019a0 con 0x7f5e1c034850 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e29e99640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e29e99640 4 set_mon_vals no callback set 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 1 -- 172.21.15.94:0/4243527664 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5e10001d30 con 0x7f5e1c034850 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.377+0000 7f5e237fe640 10 monclient: dump: 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.366 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.378+0000 7f5e29e99640 10 monclient: shutdown 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.378+0000 7f5e29e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.378+0000 7f5e29e99640 1 -- 172.21.15.94:0/4243527664 >> v1:172.21.15.94:6789/0 conn(0x7f5e1c034850 legacy=0x7f5e1c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:34.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.381+0000 7f5e29e99640 1 -- 172.21.15.94:0/4243527664 shutdown_connections 2024-03-29T10:40:34.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.381+0000 7f5e29e99640 1 -- 172.21.15.94:0/4243527664 >> 172.21.15.94:0/4243527664 conn(0x7f5e1c0058f0 msgr2=0x7f5e1c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:34.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.384+0000 7f5e29e99640 1 -- 172.21.15.94:0/4243527664 shutdown_connections 2024-03-29T10:40:34.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.384+0000 7f5e29e99640 1 -- 172.21.15.94:0/4243527664 wait complete. 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.385+0000 7f5e29e99640 10 monclient: build_initial_monmap 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.385+0000 7f5e29e99640 10 monclient: monmap: 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:34.370 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:34.371 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:34.371 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.386+0000 7f5e29e99640 1 librados: starting msgr at 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.386+0000 7f5e29e99640 1 librados: starting objecter 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.386+0000 7f5e29e99640 1 Processor -- start 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.386+0000 7f5e29e99640 1 -- start start 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.386+0000 7f5e29e99640 1 librados: setting wanted keys 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.386+0000 7f5e29e99640 1 librados: calling monclient init 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.386+0000 7f5e29e99640 10 monclient: init 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.387+0000 7f5e29e99640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:34.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.387+0000 7f5e29e99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:34.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.387+0000 7f5e29e99640 10 monclient(hunting): picked mon.noname-a con 0x7f5e1c02aac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:34.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.388+0000 7f5e29e99640 10 monclient(hunting): picked mon.noname-b con 0x7f5e1c034850 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:34.376 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.391+0000 7f5e29e99640 10 monclient(hunting): picked mon.noname-c con 0x7f5e1c0c9550 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:34.376 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.391+0000 7f5e29e99640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c0d8380 con 0x7f5e1c02aac0 2024-03-29T10:40:34.376 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.391+0000 7f5e29e99640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c02e460 con 0x7f5e1c034850 2024-03-29T10:40:34.376 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.391+0000 7f5e29e99640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c02ca40 con 0x7f5e1c0c9550 2024-03-29T10:40:34.377 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.391+0000 7f5e29e99640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:34.377 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.391+0000 7f5e29e99640 10 monclient(hunting): authenticate will time out at 885.661070s 2024-03-29T10:40:34.377 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e29698640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f5e1c0c9550 0x7f5e1c0bd0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41277/0 (socket says 172.21.15.94:41277) 2024-03-29T10:40:34.377 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e29698640 1 -- 172.21.15.94:0/3148590864 learned_addr learned my addr 172.21.15.94:0/3148590864 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 10 client.?.objecter ms_handle_connect 0x7f5e1c0c9550 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769971136 0 0) 0x7f5e1c02ca40 con 0x7f5e1c0c9550 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 10 monclient(hunting): _negotiate 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:40:34.378 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.392+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e08004eb0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.394+0000 7f5e217fa640 10 client.?.objecter ms_handle_connect 0x7f5e1c02aac0 2024-03-29T10:40:34.379 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.394+0000 7f5e217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:34.383 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.395+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1047973553 0 0) 0x7f5e08004eb0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.383 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e1c02ca40 con 0x7f5e1c0c9550 2024-03-29T10:40:34.383 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5e180018e0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.383 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:34.383 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:34.383 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 10 monclient(hunting): dump: 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4041170890 0 0) 0x7f5e1c02ca40 con 0x7f5e1c0c9550 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 >> v1:172.21.15.173:6789/0 conn(0x7f5e1c034850 legacy=0x7f5e1c0be9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 >> v1:172.21.15.94:6789/0 conn(0x7f5e1c02aac0 legacy=0x7f5e1c02eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:34.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 10 monclient: _finish_hunting 0 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 1 monclient: found mon.c 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.396+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c02e6d0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e217fa640 10 monclient: _finish_auth 0 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e217fa640 10 monclient: _check_auth_tickets 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 10 monclient: _renew_subs 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c02d440 con 0x7f5e1c0c9550 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 10 monclient: _renew_subs 2024-03-29T10:40:34.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c032690 con 0x7f5e1c0c9550 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e29e99640 1 librados: init done 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e18001ab0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.397+0000 7f5e227fc640 4 set_mon_vals no callback set 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.398+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5e180038a0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.398+0000 7f5e217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.398+0000 7f5e217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.398+0000 7f5e217fa640 10 monclient: dump: 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:34.386 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.399+0000 7f5e29e99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.399+0000 7f5e29e99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.399+0000 7f5e29e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:34.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.399+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c004cd0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.400+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f5e18003a00 con 0x7f5e1c0c9550 2024-03-29T10:40:34.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.401+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f5e180849d0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.401+0000 7f5e217fa640 10 client.4298.objecter ms_dispatch 0x7f5e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:34.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.401+0000 7f5e217fa640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:34.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.401+0000 7f5e28e97640 10 monclient: get_auth_request con 0x7f5e0806dcd0 auth_method 0 2024-03-29T10:40:34.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.401+0000 7f5e217fa640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:34.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.401+0000 7f5e217fa640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T10:40:34.389 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.402+0000 7f5e217fa640 10 client.4298.objecter ms_handle_connect 0x7f5e0806dcd0 2024-03-29T10:40:34.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.410+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e180500e0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.410+0000 7f5e217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:34.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.410+0000 7f5e217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:34.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.736+0000 7f5e29e99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:34.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.736+0000 7f5e29e99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:34.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.736+0000 7f5e29e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:34.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.736+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e1c0008d0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.738+0000 7f5e217fa640 1 -- 172.21.15.94:0/3148590864 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f5e180521e0 con 0x7f5e1c0c9550 2024-03-29T10:40:34.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.738+0000 7f5e217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:34.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.738+0000 7f5e217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:34.723 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:34.723 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:34.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 10 librados: watch_flush enter 2024-03-29T10:40:34.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 10 librados: watch_flush exit 2024-03-29T10:40:34.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T10:40:34.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 10 client.4298.objecter successfully canceled tick 2024-03-29T10:40:34.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 >> v1:172.21.15.94:6816/3829934808 conn(0x7f5e0806dcd0 legacy=0x7f5e08070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:34.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 10 monclient: shutdown 2024-03-29T10:40:34.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:34.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 >> v1:172.21.15.94:6790/0 conn(0x7f5e1c0c9550 legacy=0x7f5e1c0bd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:34.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 shutdown_connections 2024-03-29T10:40:34.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 >> 172.21.15.94:0/3148590864 conn(0x7f5e1c0058f0 msgr2=0x7f5e1c0c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:34.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.742+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 shutdown_connections 2024-03-29T10:40:34.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.743+0000 7f5e29e99640 1 -- 172.21.15.94:0/3148590864 wait complete. 2024-03-29T10:40:34.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:34.743+0000 7f5e29e99640 1 librados: shutdown 2024-03-29T10:40:34.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:41.553 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:41.706 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.721+0000 7fd1849d4640 10 monclient: get_monmap_and_config 2024-03-29T10:40:41.706 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.721+0000 7fd1849d4640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.706 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.721+0000 7fd1849d4640 10 monclient: monmap: 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:41.723606+0000 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:41.723606+0000 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:41.707 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:41.709 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.724+0000 7fd1849d4640 1 Processor -- start 2024-03-29T10:40:41.709 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.724+0000 7fd1849d4640 1 -- start start 2024-03-29T10:40:41.709 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.724+0000 7fd1849d4640 10 monclient: init 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.725+0000 7fd1849d4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.725+0000 7fd1849d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.725+0000 7fd1849d4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1800b0770 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.725+0000 7fd1849d4640 10 monclient(hunting): picked mon.noname-a con 0x7fd18014a700 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.726+0000 7fd1849d4640 10 monclient(hunting): picked mon.noname-b con 0x7fd1800ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.726+0000 7fd1849d4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1800aba70 con 0x7fd18014a700 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.726+0000 7fd1849d4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1800aecb0 con 0x7fd1800b0770 2024-03-29T10:40:41.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.726+0000 7fd1849d4640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1800af150 con 0x7fd1800ac5a0 2024-03-29T10:40:41.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.726+0000 7fd1849d4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.726+0000 7fd1849d4640 10 monclient(hunting): authenticate will time out at 892.995130s 2024-03-29T10:40:41.714 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.729+0000 7fd17e59c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fd1800b0770 0x7fd180149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:52987/0 (socket says 172.21.15.94:52987) 2024-03-29T10:40:41.714 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.729+0000 7fd17ed9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fd1800ac5a0 0x7fd180151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:52951/0 (socket says 172.21.15.94:52951) 2024-03-29T10:40:41.714 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.729+0000 7fd17ed9d640 1 -- 172.21.15.94:0/188682372 learned_addr learned my addr 172.21.15.94:0/188682372 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:41.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982346574 0 0) 0x7fd1800aecb0 con 0x7fd1800b0770 2024-03-29T10:40:41.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:40:41.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd168003520 con 0x7fd1800b0770 2024-03-29T10:40:41.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204979478 0 0) 0x7fd1800af150 con 0x7fd1800ac5a0 2024-03-29T10:40:41.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:40:41.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.730+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1800aecb0 con 0x7fd1800ac5a0 2024-03-29T10:40:41.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388307337 0 0) 0x7fd1800aecb0 con 0x7fd1800ac5a0 2024-03-29T10:40:41.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 >> v1:172.21.15.173:6789/0 conn(0x7fd1800b0770 legacy=0x7fd180149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 >> v1:172.21.15.94:6789/0 conn(0x7fd18014a700 legacy=0x7fd1800abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:41.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 1 monclient: found mon.noname-b 2024-03-29T10:40:41.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1800ac9d0 con 0x7fd1800ac5a0 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd1849d4640 5 monclient: authenticate success, global_id 4301 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd1849d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd1780019f0 con 0x7fd1800ac5a0 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd17d59a640 10 monclient: dump: 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.724 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.734+0000 7fd1849d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd178001c10 con 0x7fd1800ac5a0 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd17d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd17d59a640 1 -- 172.21.15.94:0/188682372 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd178001fa0 con 0x7fd1800ac5a0 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd17d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd17d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd17d59a640 10 monclient: dump: 2024-03-29T10:40:41.725 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd1849d4640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.735+0000 7fd1849d4640 4 set_mon_vals no callback set 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.736+0000 7fd1849d4640 10 monclient: shutdown 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.736+0000 7fd1849d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.736+0000 7fd1849d4640 1 -- 172.21.15.94:0/188682372 >> v1:172.21.15.94:6790/0 conn(0x7fd1800ac5a0 legacy=0x7fd180151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.726 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.736+0000 7fd1849d4640 1 -- 172.21.15.94:0/188682372 shutdown_connections 2024-03-29T10:40:41.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.736+0000 7fd1849d4640 1 -- 172.21.15.94:0/188682372 >> 172.21.15.94:0/188682372 conn(0x7fd18006b660 msgr2=0x7fd18006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:41.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.738+0000 7fd1849d4640 1 -- 172.21.15.94:0/188682372 shutdown_connections 2024-03-29T10:40:41.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.738+0000 7fd1849d4640 1 -- 172.21.15.94:0/188682372 wait complete. 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.739+0000 7fd1849d4640 10 monclient: build_initial_monmap 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.739+0000 7fd1849d4640 10 monclient: monmap: 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:41.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.739+0000 7fd1849d4640 1 librados: starting msgr at 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.739+0000 7fd1849d4640 1 librados: starting objecter 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.740+0000 7fd1849d4640 1 Processor -- start 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 1 -- start start 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 1 librados: setting wanted keys 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 1 librados: calling monclient init 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient: init 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient(hunting): picked mon.noname-a con 0x7fd1800ac5a0 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1800b0770 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient(hunting): picked mon.noname-b con 0x7fd18014a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd180126630 con 0x7fd1800ac5a0 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd180128340 con 0x7fd18014a700 2024-03-29T10:40:41.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd180125f50 con 0x7fd1800b0770 2024-03-29T10:40:41.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:41.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.742+0000 7fd1849d4640 10 monclient(hunting): authenticate will time out at 893.012058s 2024-03-29T10:40:41.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.747+0000 7fd17dd9b640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fd1800b0770 0x7fd180126ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:53091/0 (socket says 172.21.15.94:53091) 2024-03-29T10:40:41.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.748+0000 7fd17ed9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fd18014a700 0x7fd1801e7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:53111/0 (socket says 172.21.15.94:53111) 2024-03-29T10:40:41.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.748+0000 7fd17ed9d640 1 -- 172.21.15.94:0/3671244646 learned_addr learned my addr 172.21.15.94:0/3671244646 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:41.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.748+0000 7fd16effd640 10 client.?.objecter ms_handle_connect 0x7fd18014a700 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.748+0000 7fd16effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987842420 0 0) 0x7fd180128340 con 0x7fd18014a700 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd160004eb0 con 0x7fd18014a700 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 client.?.objecter ms_handle_connect 0x7fd1800b0770 2024-03-29T10:40:41.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919932335 0 0) 0x7fd180125f50 con 0x7fd1800b0770 2024-03-29T10:40:41.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.750+0000 7fd16effd640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:40:41.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.751+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd180128340 con 0x7fd1800b0770 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.752+0000 7fd16effd640 10 client.?.objecter ms_handle_connect 0x7fd1800ac5a0 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.752+0000 7fd16effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.752+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1723844244 0 0) 0x7fd180128340 con 0x7fd1800b0770 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.752+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd180125f50 con 0x7fd1800b0770 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.752+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4179268782 0 0) 0x7fd160004eb0 con 0x7fd18014a700 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd180128340 con 0x7fd18014a700 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102519460 0 0) 0x7fd180126630 con 0x7fd1800ac5a0 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd160004eb0 con 0x7fd1800ac5a0 2024-03-29T10:40:41.739 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd174003ee0 con 0x7fd1800b0770 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): dump: 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:41.740 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd178001930 con 0x7fd18014a700 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.753+0000 7fd16effd640 10 monclient(hunting): dump: 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:41.741 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2496103827 0 0) 0x7fd180128340 con 0x7fd18014a700 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 >> v1:172.21.15.94:6790/0 conn(0x7fd1800b0770 legacy=0x7fd180126ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 >> v1:172.21.15.94:6789/0 conn(0x7fd1800ac5a0 legacy=0x7fd180144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 1 monclient: found mon.b 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1801239d0 con 0x7fd18014a700 2024-03-29T10:40:41.742 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 10 monclient: _finish_auth 0 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.755+0000 7fd16effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 5 monclient: authenticate success, global_id 4288 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 10 monclient: _renew_subs 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd180126200 con 0x7fd18014a700 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd178004e10 con 0x7fd18014a700 2024-03-29T10:40:41.743 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd16effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 10 monclient: _renew_subs 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd17cd99640 4 set_mon_vals no callback set 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1801dfc30 con 0x7fd18014a700 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd1849d4640 1 librados: init done 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd178005120 con 0x7fd18014a700 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd16effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd16effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.756+0000 7fd16effd640 10 monclient: dump: 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:41.744 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:41.745 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.759+0000 7fd1849d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.759+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fd17801b0f0 con 0x7fd18014a700 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd1849d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd1849d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1800ab560 con 0x7fd18014a700 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fd178084e90 con 0x7fd18014a700 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd16effd640 10 client.4288.objecter ms_dispatch 0x7fd180000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:41.746 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd16effd640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:41.747 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd16effd640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:41.747 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd16effd640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T10:40:41.747 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.760+0000 7fd17e59c640 10 monclient: get_auth_request con 0x7fd16006ddb0 auth_method 0 2024-03-29T10:40:41.747 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.761+0000 7fd16effd640 10 client.4288.objecter ms_handle_connect 0x7fd16006ddb0 2024-03-29T10:40:41.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.770+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd178050770 con 0x7fd18014a700 2024-03-29T10:40:41.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.770+0000 7fd16effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:41.755 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:41.770+0000 7fd16effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:42.087 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.103+0000 7fd1849d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:42.087 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.103+0000 7fd1849d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:42.088 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.103+0000 7fd1849d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:40:42.088 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.103+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd180075f10 con 0x7fd18014a700 2024-03-29T10:40:42.091 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.106+0000 7fd16effd640 1 -- 172.21.15.94:0/3671244646 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fd1780526a0 con 0x7fd18014a700 2024-03-29T10:40:42.091 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.106+0000 7fd16effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:42.091 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.106+0000 7fd16effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:42.091 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:42.091 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:42.095 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 10 librados: watch_flush enter 2024-03-29T10:40:42.095 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 10 librados: watch_flush exit 2024-03-29T10:40:42.095 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T10:40:42.095 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 10 client.4288.objecter successfully canceled tick 2024-03-29T10:40:42.095 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 >> v1:172.21.15.94:6816/3829934808 conn(0x7fd16006ddb0 legacy=0x7fd160070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.095 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 10 monclient: shutdown 2024-03-29T10:40:42.095 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:42.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 >> v1:172.21.15.173:6789/0 conn(0x7fd18014a700 legacy=0x7fd1801e7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:42.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 shutdown_connections 2024-03-29T10:40:42.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.110+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 >> 172.21.15.94:0/3671244646 conn(0x7fd18006b660 msgr2=0x7fd18006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:42.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.111+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 shutdown_connections 2024-03-29T10:40:42.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.111+0000 7fd1849d4640 1 -- 172.21.15.94:0/3671244646 wait complete. 2024-03-29T10:40:42.096 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:42.111+0000 7fd1849d4640 1 librados: shutdown 2024-03-29T10:40:42.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:48.918 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:49.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.087+0000 7f3201f65640 10 monclient: get_monmap_and_config 2024-03-29T10:40:49.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.087+0000 7f3201f65640 10 monclient: build_initial_monmap 2024-03-29T10:40:49.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.087+0000 7f3201f65640 10 monclient: monmap: 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:49.088377+0000 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:49.088377+0000 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.088+0000 7f3201f65640 1 Processor -- start 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.088+0000 7f3201f65640 1 -- start start 2024-03-29T10:40:49.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.088+0000 7f3201f65640 10 monclient: init 2024-03-29T10:40:49.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:49.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:49.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 10 monclient(hunting): picked mon.noname-c con 0x7f31fc0b4100 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:49.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 10 monclient(hunting): picked mon.noname-a con 0x7f31fc13ee90 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:49.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 10 monclient(hunting): picked mon.noname-b con 0x7f31fc0af900 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:49.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc0afd10 con 0x7f31fc13ee90 2024-03-29T10:40:49.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc0af6f0 con 0x7f31fc0b4100 2024-03-29T10:40:49.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc0aaf60 con 0x7f31fc0af900 2024-03-29T10:40:49.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:49.075 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.089+0000 7f3201f65640 10 monclient(hunting): authenticate will time out at 900.358075s 2024-03-29T10:40:49.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.092+0000 7f31fb7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f31fc0b4100 0x7f31fc0a9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:44889/0 (socket says 172.21.15.94:44889) 2024-03-29T10:40:49.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.092+0000 7f31fb7fe640 1 -- 172.21.15.94:0/2460314945 learned_addr learned my addr 172.21.15.94:0/2460314945 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:49.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.094+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642316699 0 0) 0x7f31fc0af6f0 con 0x7f31fc0b4100 2024-03-29T10:40:49.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.094+0000 7f31fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:49.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.094+0000 7f31fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:49.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.094+0000 7f31fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:49.078 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.094+0000 7f31fa7fc640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:40:49.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.094+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e4003520 con 0x7f31fc0b4100 2024-03-29T10:40:49.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.096+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217942136 0 0) 0x7f31e4003520 con 0x7f31fc0b4100 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.096+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 >> v1:172.21.15.94:6790/0 conn(0x7f31fc0af900 legacy=0x7f31fc13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 >> v1:172.21.15.94:6789/0 conn(0x7f31fc13ee90 legacy=0x7f31fc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 1 monclient: found mon.noname-c 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc13e970 con 0x7f31fc0b4100 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f3201f65640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f3201f65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:49.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f31f4001a10 con 0x7f31fc0b4100 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f31fa7fc640 10 monclient: dump: 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.097+0000 7f3201f65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:49.084 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.099+0000 7f31fa7fc640 1 -- 172.21.15.94:0/2460314945 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31f4001ca0 con 0x7f31fc0b4100 2024-03-29T10:40:49.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.099+0000 7f31fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:49.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.099+0000 7f3201f65640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:49.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.099+0000 7f3201f65640 4 set_mon_vals no callback set 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.100+0000 7f3201f65640 10 monclient: shutdown 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.100+0000 7f3201f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.100+0000 7f3201f65640 1 -- 172.21.15.94:0/2460314945 >> v1:172.21.15.173:6789/0 conn(0x7f31fc0b4100 legacy=0x7f31fc0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.100+0000 7f3201f65640 1 -- 172.21.15.94:0/2460314945 shutdown_connections 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.100+0000 7f3201f65640 1 -- 172.21.15.94:0/2460314945 >> 172.21.15.94:0/2460314945 conn(0x7f31fc0b35e0 msgr2=0x7f31fc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.100+0000 7f3201f65640 1 -- 172.21.15.94:0/2460314945 shutdown_connections 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.100+0000 7f3201f65640 1 -- 172.21.15.94:0/2460314945 wait complete. 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.101+0000 7f3201f65640 10 monclient: build_initial_monmap 2024-03-29T10:40:49.086 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.101+0000 7f3201f65640 10 monclient: monmap: 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.101+0000 7f3201f65640 1 librados: starting msgr at 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.101+0000 7f3201f65640 1 librados: starting objecter 2024-03-29T10:40:49.087 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.102+0000 7f3201f65640 1 Processor -- start 2024-03-29T10:40:49.088 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.103+0000 7f3201f65640 1 -- start start 2024-03-29T10:40:49.088 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.103+0000 7f3201f65640 1 librados: setting wanted keys 2024-03-29T10:40:49.088 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.103+0000 7f3201f65640 1 librados: calling monclient init 2024-03-29T10:40:49.088 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.103+0000 7f3201f65640 10 monclient: init 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 10 monclient(hunting): picked mon.noname-a con 0x7f31fc0af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 10 monclient(hunting): picked mon.noname-c con 0x7f31fc0b4100 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 10 monclient(hunting): picked mon.noname-b con 0x7f31fc13ee90 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc1d6690 con 0x7f31fc0af900 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc072410 con 0x7f31fc13ee90 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc1d6170 con 0x7f31fc0b4100 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:49.089 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.104+0000 7f3201f65640 10 monclient(hunting): authenticate will time out at 900.372562s 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31fb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f31fc0af900 0x7f31fc1d6b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43931/0 (socket says 172.21.15.94:43931) 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31fb7fe640 1 -- 172.21.15.94:0/3020053314 learned_addr learned my addr 172.21.15.94:0/3020053314 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31e3fff640 10 client.?.objecter ms_handle_connect 0x7f31fc0af900 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480590117 0 0) 0x7f31fc1d6690 con 0x7f31fc0af900 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.110+0000 7f31e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:40:49.098 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e8004eb0 con 0x7f31fc0af900 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 client.?.objecter ms_handle_connect 0x7f31fc13ee90 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080014845 0 0) 0x7f31fc072410 con 0x7f31fc13ee90 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31fc1d6690 con 0x7f31fc13ee90 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031579242 0 0) 0x7f31e8004eb0 con 0x7f31fc0af900 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31fc072410 con 0x7f31fc0af900 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f31f4004e60 con 0x7f31fc0af900 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.111+0000 7f31e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 10 monclient(hunting): dump: 2024-03-29T10:40:49.099 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 106032005 0 0) 0x7f31fc1d6690 con 0x7f31fc13ee90 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31e8004eb0 con 0x7f31fc13ee90 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1402287377 0 0) 0x7f31fc072410 con 0x7f31fc0af900 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 >> v1:172.21.15.94:6790/0 conn(0x7f31fc0b4100 legacy=0x7f31fc070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:49.100 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 >> v1:172.21.15.173:6789/0 conn(0x7f31fc13ee90 legacy=0x7f31fc070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 1 monclient: found mon.a 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc1d6300 con 0x7f31fc0af900 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.112+0000 7f31e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.114+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31f40052b0 con 0x7f31fc0af900 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.114+0000 7f31e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.114+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f31f4005640 con 0x7f31fc0af900 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.114+0000 7f31e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.114+0000 7f31e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.114+0000 7f31e3fff640 10 monclient: dump: 2024-03-29T10:40:49.101 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:49.102 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.114+0000 7f31f9ffb640 4 set_mon_vals no callback set 2024-03-29T10:40:49.104 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 5 monclient: authenticate success, global_id 4299 2024-03-29T10:40:49.104 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 10 monclient: _renew_subs 2024-03-29T10:40:49.104 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31fc142ba0 con 0x7f31fc0af900 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 10 monclient: _renew_subs 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31fc0b2c30 con 0x7f31fc0af900 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.115+0000 7f3201f65640 1 librados: init done 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.117+0000 7f3201f65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.117+0000 7f3201f65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.117+0000 7f3201f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.117+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31fc06dfd0 con 0x7f31fc0af900 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.118+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f31f4001e90 con 0x7f31fc0af900 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.119+0000 7f31faffd640 10 monclient: get_auth_request con 0x7f31e806ddc0 auth_method 0 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.120+0000 7f31e3fff640 10 client.4299.objecter ms_handle_connect 0x7f31e806ddc0 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.120+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f31f40850f0 con 0x7f31fc0af900 2024-03-29T10:40:49.105 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.120+0000 7f31e3fff640 10 client.4299.objecter ms_dispatch 0x7f31fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:49.106 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.120+0000 7f31e3fff640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:49.106 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.120+0000 7f31e3fff640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:49.106 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.120+0000 7f31e3fff640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T10:40:49.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.132+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31f40509d0 con 0x7f31fc0af900 2024-03-29T10:40:49.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.132+0000 7f31e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:49.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.132+0000 7f31e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:49.435 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:49.435 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:49.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.450+0000 7f3201f65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:49.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.450+0000 7f3201f65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:49.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.450+0000 7f3201f65640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:40:49.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.450+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31fc1d6490 con 0x7f31fc0af900 2024-03-29T10:40:49.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.451+0000 7f31e3fff640 1 -- 172.21.15.94:0/3020053314 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f31f4052900 con 0x7f31fc0af900 2024-03-29T10:40:49.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.451+0000 7f31e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:49.437 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.451+0000 7f31e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:49.438 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.454+0000 7f3201f65640 10 librados: watch_flush enter 2024-03-29T10:40:49.438 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.455+0000 7f3201f65640 10 librados: watch_flush exit 2024-03-29T10:40:49.438 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.455+0000 7f3201f65640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T10:40:49.438 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.455+0000 7f3201f65640 10 client.4299.objecter successfully canceled tick 2024-03-29T10:40:49.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.456+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 >> v1:172.21.15.94:6816/3829934808 conn(0x7f31e806ddc0 legacy=0x7f31e8070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:49.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.456+0000 7f3201f65640 10 monclient: shutdown 2024-03-29T10:40:49.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.456+0000 7f3201f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:49.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.456+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 >> v1:172.21.15.94:6789/0 conn(0x7f31fc0af900 legacy=0x7f31fc1d6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:49.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.458+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 shutdown_connections 2024-03-29T10:40:49.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.458+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 >> 172.21.15.94:0/3020053314 conn(0x7f31fc0b35e0 msgr2=0x7f31fc0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:49.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.458+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 shutdown_connections 2024-03-29T10:40:49.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.458+0000 7f3201f65640 1 -- 172.21.15.94:0/3020053314 wait complete. 2024-03-29T10:40:49.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:49.458+0000 7f3201f65640 1 librados: shutdown 2024-03-29T10:40:49.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:56.266 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:56.418 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.435+0000 7fd97b59e640 10 monclient: get_monmap_and_config 2024-03-29T10:40:56.418 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.435+0000 7fd97b59e640 10 monclient: build_initial_monmap 2024-03-29T10:40:56.418 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.435+0000 7fd97b59e640 10 monclient: monmap: 2024-03-29T10:40:56.418 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:56.418 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:40:56.435913+0000 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:40:56.435913+0000 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:40:56.419 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:56.420 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.436+0000 7fd97b59e640 1 Processor -- start 2024-03-29T10:40:56.420 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.436+0000 7fd97b59e640 1 -- start start 2024-03-29T10:40:56.420 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.436+0000 7fd97b59e640 10 monclient: init 2024-03-29T10:40:56.421 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.438+0000 7fd97b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:56.421 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.438+0000 7fd97b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.438+0000 7fd97b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fd9740389e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.441+0000 7fd97b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fd974038dc0 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.441+0000 7fd97b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fd974030660 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.441+0000 7fd97b59e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740c54c0 con 0x7fd9740389e0 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.441+0000 7fd97b59e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd974036430 con 0x7fd974038dc0 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.441+0000 7fd97b59e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd974039840 con 0x7fd974030660 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.441+0000 7fd97b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:56.425 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.441+0000 7fd97b59e640 10 monclient(hunting): authenticate will time out at 907.709863s 2024-03-29T10:40:56.428 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97a59c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd9740389e0 0x7fd974033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:34535/0 (socket says 172.21.15.94:34535) 2024-03-29T10:40:56.428 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97a59c640 1 -- 172.21.15.94:0/3345357548 learned_addr learned my addr 172.21.15.94:0/3345357548 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373073466 0 0) 0x7fd9740c54c0 con 0x7fd9740389e0 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97959a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97959a640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.445+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd964003520 con 0x7fd9740389e0 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.446+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 824935775 0 0) 0x7fd964003520 con 0x7fd9740389e0 2024-03-29T10:40:56.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.446+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6790/0 conn(0x7fd974030660 legacy=0x7fd9740d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.173:6789/0 conn(0x7fd974038dc0 legacy=0x7fd9740cadd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 10 monclient: _finish_hunting 0 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 1 monclient: found mon.noname-a 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974035690 con 0x7fd9740389e0 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 10 monclient: _finish_auth 0 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 10 monclient: _check_auth_tickets 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd96c001690 con 0x7fd9740389e0 2024-03-29T10:40:56.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.447+0000 7fd97959a640 10 monclient: dump: 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:56.432 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:56.433 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:56.433 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:56.433 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97a59c640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6789/0 conn(0x7fd9740389e0 legacy=0x7fd974033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97a59c640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6789/0 conn(0x7fd9740389e0 legacy=0x7fd974033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97a59c640 1 --1- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6789/0 conn(0x7fd9740389e0 0x7fd974033520 :-1 s=OPENED pgs=185600 cs=1 l=1).handle_message read tag failed 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97a59c640 1 --1- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6789/0 conn(0x7fd9740389e0 0x7fd974033520 :-1 s=OPENED pgs=185600 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97959a640 10 monclient: ms_handle_reset current mon v1:172.21.15.94:6789/0 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97959a640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6789/0 conn(0x7fd9740389e0 legacy=0x7fd974033520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97959a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:56.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.448+0000 7fd97959a640 10 monclient(hunting): picked mon.b con 0x7fd964003af0 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:56.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.449+0000 7fd97959a640 10 monclient(hunting): picked mon.c con 0x7fd974030660 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:56.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.449+0000 7fd97959a640 10 monclient(hunting): picked mon.a con 0x7fd96400b390 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:56.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.449+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd964003520 con 0x7fd96400b390 2024-03-29T10:40:56.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.449+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd974036430 con 0x7fd964003af0 2024-03-29T10:40:56.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.449+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd974039840 con 0x7fd974030660 2024-03-29T10:40:56.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.449+0000 7fd97959a640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:56.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.449+0000 7fd97959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.94:6789/0 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.453+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523381225 0 0) 0x7fd964003520 con 0x7fd96400b390 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.453+0000 7fd97959a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.453+0000 7fd97959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.453+0000 7fd97959a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.454+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 --> v1:172.21.15.94:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fd96400f210 con 0x7fd96400b390 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2116322902 0 0) 0x7fd96400f210 con 0x7fd96400b390 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6790/0 conn(0x7fd974030660 legacy=0x7fd964009a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.173:6789/0 conn(0x7fd964003af0 legacy=0x7fd964003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: _finish_hunting 0 2024-03-29T10:40:56.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 1 monclient: found mon.a 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fd9740c54c0 con 0x7fd96400b390 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: _finish_auth 0 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: _check_auth_tickets 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd970001300 con 0x7fd96400b390 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: dump: 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:56.442 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97b59e640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 1 -- 172.21.15.94:0/3345357548 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd970001460 con 0x7fd96400b390 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97b59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:56.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.457+0000 7fd97b59e640 4 set_mon_vals no callback set 2024-03-29T10:40:56.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.458+0000 7fd97b59e640 10 monclient: shutdown 2024-03-29T10:40:56.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.459+0000 7fd97b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:56.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.459+0000 7fd97b59e640 1 -- 172.21.15.94:0/3345357548 >> v1:172.21.15.94:6789/0 conn(0x7fd96400b390 legacy=0x7fd96400d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:56.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.459+0000 7fd97b59e640 1 -- 172.21.15.94:0/3345357548 shutdown_connections 2024-03-29T10:40:56.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.459+0000 7fd97b59e640 1 -- 172.21.15.94:0/3345357548 >> 172.21.15.94:0/3345357548 conn(0x7fd974005c00 msgr2=0x7fd974006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:56.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.461+0000 7fd97b59e640 1 -- 172.21.15.94:0/3345357548 shutdown_connections 2024-03-29T10:40:56.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.461+0000 7fd97b59e640 1 -- 172.21.15.94:0/3345357548 wait complete. 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.462+0000 7fd97b59e640 10 monclient: build_initial_monmap 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.462+0000 7fd97b59e640 10 monclient: monmap: 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:40:56.446 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:56.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.463+0000 7fd97b59e640 1 librados: starting msgr at 2024-03-29T10:40:56.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.463+0000 7fd97b59e640 1 librados: starting objecter 2024-03-29T10:40:56.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.463+0000 7fd97b59e640 1 Processor -- start 2024-03-29T10:40:56.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.463+0000 7fd97b59e640 1 -- start start 2024-03-29T10:40:56.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.464+0000 7fd97b59e640 1 librados: setting wanted keys 2024-03-29T10:40:56.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.464+0000 7fd97b59e640 1 librados: calling monclient init 2024-03-29T10:40:56.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.464+0000 7fd97b59e640 10 monclient: init 2024-03-29T10:40:56.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.464+0000 7fd97b59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:56.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.465+0000 7fd97b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:56.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.465+0000 7fd97b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fd964003af0 addr v1:172.21.15.94:6789/0 2024-03-29T10:40:56.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.465+0000 7fd97b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fd96400b390 addr v1:172.21.15.173:6789/0 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.467+0000 7fd97b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fd974030660 addr v1:172.21.15.94:6790/0 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.467+0000 7fd97ad9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fd974030660 0x7fd9740c2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:52467/0 (socket says 172.21.15.94:52467) 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.467+0000 7fd97ad9d640 1 -- 172.21.15.94:0/3441273390 learned_addr learned my addr 172.21.15.94:0/3441273390 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97402f410 con 0x7fd964003af0 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740c6b40 con 0x7fd96400b390 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97402f0a0 con 0x7fd974030660 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd97b59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:56.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd97b59e640 10 monclient(hunting): authenticate will time out at 907.736539s 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 10 client.?.objecter ms_handle_connect 0x7fd974030660 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664690110 0 0) 0x7fd97402f0a0 con 0x7fd974030660 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:40:56.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.468+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c004eb0 con 0x7fd974030660 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593057076 0 0) 0x7fd95c004eb0 con 0x7fd974030660 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd97402f0a0 con 0x7fd974030660 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd970001870 con 0x7fd974030660 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 10 monclient(hunting): dump: 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:56.454 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:56.455 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:56.455 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:56.455 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:56.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:56.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2822166058 0 0) 0x7fd97402f0a0 con 0x7fd974030660 2024-03-29T10:40:56.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.470+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 >> v1:172.21.15.173:6789/0 conn(0x7fd96400b390 legacy=0x7fd97415f850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 >> v1:172.21.15.94:6789/0 conn(0x7fd964003af0 legacy=0x7fd97402fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 1 monclient: found mon.c 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97402f8a0 con 0x7fd974030660 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 10 monclient: _finish_auth 0 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 10 client.?.objecter ms_handle_connect 0x7fd964003af0 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd96affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:56.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 5 monclient: authenticate success, global_id 4307 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 10 monclient: _renew_subs 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97415ff60 con 0x7fd974030660 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 10 monclient: _renew_subs 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9740c2b30 con 0x7fd974030660 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.472+0000 7fd97b59e640 1 librados: init done 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.474+0000 7fd97b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:56.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.474+0000 7fd97b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:56.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.474+0000 7fd97b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:56.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.474+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd974007b90 con 0x7fd974030660 2024-03-29T10:40:56.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.477+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9700019d0 con 0x7fd974030660 2024-03-29T10:40:56.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.478+0000 7fd96affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:56.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.478+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd970003b50 con 0x7fd974030660 2024-03-29T10:40:56.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.478+0000 7fd96affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.478+0000 7fd96affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.478+0000 7fd96affd640 10 monclient: dump: 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.479+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fd970003d70 con 0x7fd974030660 2024-03-29T10:40:56.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.479+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fd970084bf0 con 0x7fd974030660 2024-03-29T10:40:56.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.479+0000 7fd96affd640 10 client.4307.objecter ms_dispatch 0x7fd97c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:56.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.479+0000 7fd96affd640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:56.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.479+0000 7fd96affd640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:56.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.480+0000 7fd96affd640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-29T10:40:56.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.480+0000 7fd96bfff640 4 set_mon_vals no callback set 2024-03-29T10:40:56.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.480+0000 7fd97a59c640 10 monclient: get_auth_request con 0x7fd95c06db30 auth_method 0 2024-03-29T10:40:56.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.480+0000 7fd96affd640 10 client.4307.objecter ms_handle_connect 0x7fd95c06db30 2024-03-29T10:40:56.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.483+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd970050460 con 0x7fd974030660 2024-03-29T10:40:56.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.484+0000 7fd96affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:56.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.484+0000 7fd96affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:56.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.814+0000 7fd97b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:56.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.814+0000 7fd97b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:56.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.814+0000 7fd97b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:40:56.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.814+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd97402f230 con 0x7fd974030660 2024-03-29T10:40:56.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.819+0000 7fd96affd640 1 -- 172.21.15.94:0/3441273390 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fd9700524c0 con 0x7fd974030660 2024-03-29T10:40:56.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.819+0000 7fd96affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:56.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.819+0000 7fd96affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:56.804 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:40:56.804 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:56.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 10 librados: watch_flush enter 2024-03-29T10:40:56.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 10 librados: watch_flush exit 2024-03-29T10:40:56.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-29T10:40:56.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 10 client.4307.objecter successfully canceled tick 2024-03-29T10:40:56.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 >> v1:172.21.15.94:6816/3829934808 conn(0x7fd95c06db30 legacy=0x7fd95c06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:56.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 10 monclient: shutdown 2024-03-29T10:40:56.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:56.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 >> v1:172.21.15.94:6790/0 conn(0x7fd974030660 legacy=0x7fd9740c2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:56.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 shutdown_connections 2024-03-29T10:40:56.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 >> 172.21.15.94:0/3441273390 conn(0x7fd974005c00 msgr2=0x7fd9740347e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:56.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 shutdown_connections 2024-03-29T10:40:56.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 1 -- 172.21.15.94:0/3441273390 wait complete. 2024-03-29T10:40:56.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:40:56.824+0000 7fd97b59e640 1 librados: shutdown 2024-03-29T10:40:56.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:03.631 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:03.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.800+0000 7fca59c25640 10 monclient: get_monmap_and_config 2024-03-29T10:41:03.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.800+0000 7fca59c25640 10 monclient: build_initial_monmap 2024-03-29T10:41:03.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.800+0000 7fca59c25640 10 monclient: monmap: 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:03.800957+0000 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:03.800957+0000 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:03.784 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:03.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.801+0000 7fca59c25640 1 Processor -- start 2024-03-29T10:41:03.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.801+0000 7fca59c25640 1 -- start start 2024-03-29T10:41:03.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.801+0000 7fca59c25640 10 monclient: init 2024-03-29T10:41:03.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.802+0000 7fca59c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:03.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.802+0000 7fca59c25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:03.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.802+0000 7fca59c25640 10 monclient(hunting): picked mon.noname-a con 0x7fca541457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:03.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.802+0000 7fca59c25640 10 monclient(hunting): picked mon.noname-b con 0x7fca5413f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:03.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.806+0000 7fca59c25640 10 monclient(hunting): picked mon.noname-c con 0x7fca540a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:03.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.806+0000 7fca59c25640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca540aaef0 con 0x7fca541457f0 2024-03-29T10:41:03.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.806+0000 7fca59c25640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca54145bd0 con 0x7fca540a9d70 2024-03-29T10:41:03.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.806+0000 7fca537fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fca541457f0 0x7fca5413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49485/0 (socket says 172.21.15.94:49485) 2024-03-29T10:41:03.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.806+0000 7fca537fe640 1 -- 172.21.15.94:0/693052223 learned_addr learned my addr 172.21.15.94:0/693052223 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:03.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.806+0000 7fca59c25640 1 -- 172.21.15.94:0/693052223 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca54145eb0 con 0x7fca5413f650 2024-03-29T10:41:03.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca59c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:03.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca59c25640 10 monclient(hunting): authenticate will time out at 915.075272s 2024-03-29T10:41:03.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888386814 0 0) 0x7fca540aaef0 con 0x7fca541457f0 2024-03-29T10:41:03.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:03.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:03.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:03.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca527fc640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:41:03.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.807+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca3c003520 con 0x7fca541457f0 2024-03-29T10:41:03.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2469824926 0 0) 0x7fca3c003520 con 0x7fca541457f0 2024-03-29T10:41:03.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 >> v1:172.21.15.94:6790/0 conn(0x7fca5413f650 legacy=0x7fca540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 >> v1:172.21.15.173:6789/0 conn(0x7fca540a9d70 legacy=0x7fca5414d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 1 monclient: found mon.noname-a 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca54146340 con 0x7fca541457f0 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fca44001670 con 0x7fca541457f0 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:03.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.810+0000 7fca527fc640 10 monclient: dump: 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.811+0000 7fca59c25640 5 monclient: authenticate success, global_id 4308 2024-03-29T10:41:03.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.811+0000 7fca59c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:03.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca44001840 con 0x7fca541457f0 2024-03-29T10:41:03.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca527fc640 1 -- 172.21.15.94:0/693052223 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fca44001c50 con 0x7fca541457f0 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca527fc640 10 monclient: dump: 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:03.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca59c25640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:03.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.813+0000 7fca59c25640 4 set_mon_vals no callback set 2024-03-29T10:41:03.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.814+0000 7fca59c25640 10 monclient: shutdown 2024-03-29T10:41:03.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.814+0000 7fca59c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:03.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.814+0000 7fca59c25640 1 -- 172.21.15.94:0/693052223 >> v1:172.21.15.94:6789/0 conn(0x7fca541457f0 legacy=0x7fca5413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:03.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.814+0000 7fca59c25640 1 -- 172.21.15.94:0/693052223 shutdown_connections 2024-03-29T10:41:03.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.814+0000 7fca59c25640 1 -- 172.21.15.94:0/693052223 >> 172.21.15.94:0/693052223 conn(0x7fca54066a00 msgr2=0x7fca54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:03.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.814+0000 7fca59c25640 1 -- 172.21.15.94:0/693052223 shutdown_connections 2024-03-29T10:41:03.802 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.817+0000 7fca59c25640 1 -- 172.21.15.94:0/693052223 wait complete. 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.818+0000 7fca59c25640 10 monclient: build_initial_monmap 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.818+0000 7fca59c25640 10 monclient: monmap: 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:03.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.818+0000 7fca59c25640 1 librados: starting msgr at 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.818+0000 7fca59c25640 1 librados: starting objecter 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.819+0000 7fca59c25640 1 Processor -- start 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.819+0000 7fca59c25640 1 -- start start 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.819+0000 7fca59c25640 1 librados: setting wanted keys 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.819+0000 7fca59c25640 1 librados: calling monclient init 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.819+0000 7fca59c25640 10 monclient: init 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.820+0000 7fca59c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:03.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.820+0000 7fca59c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.820+0000 7fca59c25640 10 monclient(hunting): picked mon.noname-a con 0x7fca540a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.822+0000 7fca59c25640 10 monclient(hunting): picked mon.noname-b con 0x7fca5413f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.822+0000 7fca59c25640 10 monclient(hunting): picked mon.noname-c con 0x7fca541457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.822+0000 7fca59c25640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca541d8210 con 0x7fca540a9d70 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.822+0000 7fca59c25640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca540af5a0 con 0x7fca5413f650 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.822+0000 7fca59c25640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca541d7e50 con 0x7fca541457f0 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.822+0000 7fca59c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:03.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.822+0000 7fca59c25640 10 monclient(hunting): authenticate will time out at 915.090626s 2024-03-29T10:41:03.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.824+0000 7fca537fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fca540a9d70 0x7fca541d7740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49639/0 (socket says 172.21.15.94:49639) 2024-03-29T10:41:03.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.824+0000 7fca537fe640 1 -- 172.21.15.94:0/934219445 learned_addr learned my addr 172.21.15.94:0/934219445 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:03.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 10 client.?.objecter ms_handle_connect 0x7fca5413f650 2024-03-29T10:41:03.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:03.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89741194 0 0) 0x7fca540af5a0 con 0x7fca5413f650 2024-03-29T10:41:03.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:03.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:03.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:03.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:41:03.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.826+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca40004eb0 con 0x7fca5413f650 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 10 client.?.objecter ms_handle_connect 0x7fca540a9d70 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566686840 0 0) 0x7fca40004eb0 con 0x7fca5413f650 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca540af5a0 con 0x7fca5413f650 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fca480018f0 con 0x7fca5413f650 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 10 monclient(hunting): dump: 2024-03-29T10:41:03.817 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.828+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613341539 0 0) 0x7fca54146340 con 0x7fca540a9d70 2024-03-29T10:41:03.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca40004eb0 con 0x7fca540a9d70 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 client.?.objecter ms_handle_connect 0x7fca541457f0 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405432553 0 0) 0x7fca541d7e50 con 0x7fca541457f0 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.829+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54146340 con 0x7fca541457f0 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.830+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2273965873 0 0) 0x7fca54146340 con 0x7fca541457f0 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.830+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca541d7e50 con 0x7fca541457f0 2024-03-29T10:41:03.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.830+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fca4c001a70 con 0x7fca541457f0 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.830+0000 7fca3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.830+0000 7fca3bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.830+0000 7fca3bfff640 10 monclient(hunting): dump: 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3882659404 0 0) 0x7fca540af5a0 con 0x7fca5413f650 2024-03-29T10:41:03.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 >> v1:172.21.15.94:6790/0 conn(0x7fca541457f0 legacy=0x7fca540ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 >> v1:172.21.15.94:6789/0 conn(0x7fca540a9d70 legacy=0x7fca541d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 1 monclient: found mon.b 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca541d7fe0 con 0x7fca5413f650 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 10 monclient: _finish_auth 0 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 5 monclient: authenticate success, global_id 4297 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 10 monclient: _renew_subs 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca54142070 con 0x7fca5413f650 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:03.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 10 monclient: _renew_subs 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca540b14a0 con 0x7fca5413f650 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca59c25640 1 librados: init done 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca48001ac0 con 0x7fca5413f650 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca517fa640 4 set_mon_vals no callback set 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.831+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fca480038c0 con 0x7fca5413f650 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.832+0000 7fca3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.832+0000 7fca3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.832+0000 7fca3bfff640 10 monclient: dump: 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.822 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.832+0000 7fca59c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:03.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.832+0000 7fca59c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:03.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.832+0000 7fca59c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:03.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.832+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca34005080 con 0x7fca5413f650 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.838+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fca48003a20 con 0x7fca5413f650 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.839+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fca48083890 con 0x7fca5413f650 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.839+0000 7fca3bfff640 10 client.4297.objecter ms_dispatch 0x7fca54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.839+0000 7fca3bfff640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.839+0000 7fca3bfff640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.839+0000 7fca3bfff640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.840+0000 7fca537fe640 10 monclient: get_auth_request con 0x7fca4006dfb0 auth_method 0 2024-03-29T10:41:03.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.841+0000 7fca3bfff640 10 client.4297.objecter ms_handle_connect 0x7fca4006dfb0 2024-03-29T10:41:03.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.844+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca4804efa0 con 0x7fca5413f650 2024-03-29T10:41:03.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.844+0000 7fca3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:03.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:03.844+0000 7fca3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:04.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.186+0000 7fca59c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:04.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.186+0000 7fca59c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:04.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.186+0000 7fca59c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:04.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.186+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca34005600 con 0x7fca5413f650 2024-03-29T10:41:04.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.187+0000 7fca3bfff640 1 -- 172.21.15.94:0/934219445 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fca480510a0 con 0x7fca5413f650 2024-03-29T10:41:04.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.187+0000 7fca3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:04.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.187+0000 7fca3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:04.172 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:04.172 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:04.176 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.191+0000 7fca59c25640 10 librados: watch_flush enter 2024-03-29T10:41:04.176 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.192+0000 7fca59c25640 10 librados: watch_flush exit 2024-03-29T10:41:04.176 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.192+0000 7fca59c25640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-29T10:41:04.176 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.192+0000 7fca59c25640 10 client.4297.objecter successfully canceled tick 2024-03-29T10:41:04.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.192+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 >> v1:172.21.15.94:6816/3829934808 conn(0x7fca4006dfb0 legacy=0x7fca40070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:04.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.192+0000 7fca59c25640 10 monclient: shutdown 2024-03-29T10:41:04.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.192+0000 7fca59c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:04.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.192+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 >> v1:172.21.15.173:6789/0 conn(0x7fca5413f650 legacy=0x7fca540b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:04.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.193+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 shutdown_connections 2024-03-29T10:41:04.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.193+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 >> 172.21.15.94:0/934219445 conn(0x7fca54066a00 msgr2=0x7fca5413ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:04.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.194+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 shutdown_connections 2024-03-29T10:41:04.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.194+0000 7fca59c25640 1 -- 172.21.15.94:0/934219445 wait complete. 2024-03-29T10:41:04.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:04.194+0000 7fca59c25640 1 librados: shutdown 2024-03-29T10:41:04.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:11.003 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:11.162 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.174+0000 7f27c47d7640 10 monclient: get_monmap_and_config 2024-03-29T10:41:11.162 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.174+0000 7f27c47d7640 10 monclient: build_initial_monmap 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.174+0000 7f27c47d7640 10 monclient: monmap: 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:11.175253+0000 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:11.175253+0000 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.179+0000 7f27c47d7640 1 Processor -- start 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.179+0000 7f27c47d7640 1 -- start start 2024-03-29T10:41:11.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.179+0000 7f27c47d7640 10 monclient: init 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.180+0000 7f27c47d7640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.180+0000 7f27c47d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.180+0000 7f27c47d7640 10 monclient(hunting): picked mon.noname-a con 0x7f27bc0a9930 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.180+0000 7f27c47d7640 10 monclient(hunting): picked mon.noname-b con 0x7f27bc149dc0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c47d7640 10 monclient(hunting): picked mon.noname-c con 0x7f27bc14ff40 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c47d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0aae60 con 0x7f27bc0a9930 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c47d7640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0b4310 con 0x7f27bc14ff40 2024-03-29T10:41:11.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c47d7640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0afdd0 con 0x7f27bc149dc0 2024-03-29T10:41:11.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c47d7640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:11.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c47d7640 10 monclient(hunting): authenticate will time out at 922.450265s 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c1d72640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f27bc149dc0 0x7f27bc14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:57218/0 (socket says 172.21.15.94:57218) 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.182+0000 7f27c1d72640 1 -- 172.21.15.94:0/2359957356 learned_addr learned my addr 172.21.15.94:0/2359957356 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754853856 0 0) 0x7f27bc0afdd0 con 0x7f27bc149dc0 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 10 monclient(hunting): _negotiate 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a8003520 con 0x7f27bc149dc0 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154007147 0 0) 0x7f27a8003520 con 0x7f27bc149dc0 2024-03-29T10:41:11.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.183+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 >> v1:172.21.15.173:6789/0 conn(0x7f27bc14ff40 legacy=0x7f27bc152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:11.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.186+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 >> v1:172.21.15.94:6789/0 conn(0x7f27bc0a9930 legacy=0x7f27bc143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:11.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: _finish_hunting 0 2024-03-29T10:41:11.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 1 monclient: found mon.noname-b 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc14a950 con 0x7f27bc149dc0 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: _finish_auth 0 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: _check_auth_tickets 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c47d7640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c47d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f27ac001690 con 0x7f27bc149dc0 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: dump: 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:11.172 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27ac001920 con 0x7f27bc149dc0 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 1 -- 172.21.15.94:0/2359957356 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f27ac001cb0 con 0x7f27bc149dc0 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:11.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c1571640 10 monclient: dump: 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c47d7640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:11.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.187+0000 7f27c47d7640 4 set_mon_vals no callback set 2024-03-29T10:41:11.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.188+0000 7f27c47d7640 10 monclient: shutdown 2024-03-29T10:41:11.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.188+0000 7f27c47d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:11.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.188+0000 7f27c47d7640 1 -- 172.21.15.94:0/2359957356 >> v1:172.21.15.94:6790/0 conn(0x7f27bc149dc0 legacy=0x7f27bc14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:11.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.188+0000 7f27c47d7640 1 -- 172.21.15.94:0/2359957356 shutdown_connections 2024-03-29T10:41:11.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.188+0000 7f27c47d7640 1 -- 172.21.15.94:0/2359957356 >> 172.21.15.94:0/2359957356 conn(0x7f27bc0b35e0 msgr2=0x7f27bc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:11.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.189+0000 7f27c47d7640 1 -- 172.21.15.94:0/2359957356 shutdown_connections 2024-03-29T10:41:11.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.189+0000 7f27c47d7640 1 -- 172.21.15.94:0/2359957356 wait complete. 2024-03-29T10:41:11.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.189+0000 7f27c47d7640 10 monclient: build_initial_monmap 2024-03-29T10:41:11.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.189+0000 7f27c47d7640 10 monclient: monmap: 2024-03-29T10:41:11.178 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.190+0000 7f27c47d7640 1 librados: starting msgr at 2024-03-29T10:41:11.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.190+0000 7f27c47d7640 1 librados: starting objecter 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.190+0000 7f27c47d7640 1 Processor -- start 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.190+0000 7f27c47d7640 1 -- start start 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.190+0000 7f27c47d7640 1 librados: setting wanted keys 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.190+0000 7f27c47d7640 1 librados: calling monclient init 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.190+0000 7f27c47d7640 10 monclient: init 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.191+0000 7f27c47d7640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.191+0000 7f27c47d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:11.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.191+0000 7f27c47d7640 10 monclient(hunting): picked mon.noname-c con 0x7f27bc0a9930 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:11.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.197+0000 7f27c47d7640 10 monclient(hunting): picked mon.noname-b con 0x7f27bc14ff40 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:11.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c2573640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f27bc0a9930 0x7f27bc138660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:49758/0 (socket says 172.21.15.94:49758) 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c47d7640 10 monclient(hunting): picked mon.noname-a con 0x7f27bc13ffb0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c47d7640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc13fa90 con 0x7f27bc13ffb0 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c47d7640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc13b120 con 0x7f27bc14ff40 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c47d7640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc13b2b0 con 0x7f27bc0a9930 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c47d7640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c47d7640 10 monclient(hunting): authenticate will time out at 922.467801s 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27c2573640 1 -- 172.21.15.94:0/3660405762 learned_addr learned my addr 172.21.15.94:0/3660405762 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.199+0000 7f27b2ffd640 10 client.?.objecter ms_handle_connect 0x7f27bc0a9930 2024-03-29T10:41:11.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.200+0000 7f27b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:11.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.200+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921721047 0 0) 0x7f27bc13b2b0 con 0x7f27bc0a9930 2024-03-29T10:41:11.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.200+0000 7f27b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:11.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.200+0000 7f27b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:11.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.200+0000 7f27b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:11.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.200+0000 7f27b2ffd640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:41:11.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.200+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a0004eb0 con 0x7f27bc0a9930 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 client.?.objecter ms_handle_connect 0x7f27bc13ffb0 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3660118726 0 0) 0x7f27a0004eb0 con 0x7f27bc0a9930 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27bc13b2b0 con 0x7f27bc0a9930 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f27b8001960 con 0x7f27bc0a9930 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:11.185 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556006311 0 0) 0x7f27bc13fa90 con 0x7f27bc13ffb0 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a0004eb0 con 0x7f27bc13ffb0 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 client.?.objecter ms_handle_connect 0x7f27bc14ff40 2024-03-29T10:41:11.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.201+0000 7f27b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:11.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3727220626 0 0) 0x7f27a0004eb0 con 0x7f27bc13ffb0 2024-03-29T10:41:11.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27bc13fa90 con 0x7f27bc13ffb0 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4064231600 0 0) 0x7f27bc13b2b0 con 0x7f27bc0a9930 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 >> v1:172.21.15.173:6789/0 conn(0x7f27bc14ff40 legacy=0x7f27bc148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 >> v1:172.21.15.94:6789/0 conn(0x7f27bc13ffb0 legacy=0x7f27bc13e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 1 monclient: found mon.c 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc13e750 con 0x7f27bc0a9930 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 10 monclient: _renew_subs 2024-03-29T10:41:11.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc2010b0 con 0x7f27bc0a9930 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 10 monclient: _renew_subs 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc201420 con 0x7f27bc0a9930 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.202+0000 7f27c47d7640 1 librados: init done 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.203+0000 7f27c47d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.203+0000 7f27c47d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.203+0000 7f27c47d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:11.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.203+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc13f8f0 con 0x7f27bc0a9930 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27b8003610 con 0x7f27bc0a9930 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f27b8003960 con 0x7f27bc0a9930 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27b2ffd640 10 monclient: dump: 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:11.192 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:11.193 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:11.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f27b8003b80 con 0x7f27bc0a9930 2024-03-29T10:41:11.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27c0d70640 4 set_mon_vals no callback set 2024-03-29T10:41:11.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.206+0000 7f27c1d72640 10 monclient: get_auth_request con 0x7f27a006def0 auth_method 0 2024-03-29T10:41:11.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.207+0000 7f27b2ffd640 10 client.4319.objecter ms_handle_connect 0x7f27a006def0 2024-03-29T10:41:11.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.207+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f27b8084b60 con 0x7f27bc0a9930 2024-03-29T10:41:11.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.207+0000 7f27b2ffd640 10 client.4319.objecter ms_dispatch 0x7f27bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:11.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.207+0000 7f27b2ffd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:11.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.207+0000 7f27b2ffd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:11.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.207+0000 7f27b2ffd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T10:41:11.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.212+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27bc13f8f0 con 0x7f27bc0a9930 2024-03-29T10:41:11.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.212+0000 7f27b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:11.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.212+0000 7f27b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:11.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.537+0000 7f27c47d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:11.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.537+0000 7f27c47d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:11.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.537+0000 7f27c47d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:11.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.537+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27bc13f8f0 con 0x7f27bc0a9930 2024-03-29T10:41:11.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.540+0000 7f27b2ffd640 1 -- 172.21.15.94:0/3660405762 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f27bc13f8f0 con 0x7f27bc0a9930 2024-03-29T10:41:11.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.540+0000 7f27b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:11.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.540+0000 7f27b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:11.525 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:11.525 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:11.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.545+0000 7f27c47d7640 10 librados: watch_flush enter 2024-03-29T10:41:11.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.545+0000 7f27c47d7640 10 librados: watch_flush exit 2024-03-29T10:41:11.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.545+0000 7f27c47d7640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T10:41:11.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.545+0000 7f27c47d7640 10 client.4319.objecter successfully canceled tick 2024-03-29T10:41:11.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.547+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 >> v1:172.21.15.94:6816/3829934808 conn(0x7f27a006def0 legacy=0x7f27a00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:11.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.547+0000 7f27c47d7640 10 monclient: shutdown 2024-03-29T10:41:11.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.547+0000 7f27c47d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:11.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.547+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 >> v1:172.21.15.94:6790/0 conn(0x7f27bc0a9930 legacy=0x7f27bc138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:11.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.547+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 shutdown_connections 2024-03-29T10:41:11.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.548+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 >> 172.21.15.94:0/3660405762 conn(0x7f27bc0b35e0 msgr2=0x7f27bc0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:11.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.548+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 shutdown_connections 2024-03-29T10:41:11.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.548+0000 7f27c47d7640 1 -- 172.21.15.94:0/3660405762 wait complete. 2024-03-29T10:41:11.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:11.548+0000 7f27c47d7640 1 librados: shutdown 2024-03-29T10:41:11.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:18.358 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:18.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.530+0000 7f6e330c8640 10 monclient: get_monmap_and_config 2024-03-29T10:41:18.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.530+0000 7f6e330c8640 10 monclient: build_initial_monmap 2024-03-29T10:41:18.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.530+0000 7f6e330c8640 10 monclient: monmap: 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:18.530891+0000 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:18.530891+0000 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:18.514 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:18.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.531+0000 7f6e330c8640 1 Processor -- start 2024-03-29T10:41:18.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.531+0000 7f6e330c8640 1 -- start start 2024-03-29T10:41:18.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.531+0000 7f6e330c8640 10 monclient: init 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c1457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c13f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c0aaef0 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c145bd0 con 0x7f6e2c13f650 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c145eb0 con 0x7f6e2c1457f0 2024-03-29T10:41:18.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e330c8640 10 monclient(hunting): authenticate will time out at 929.800389s 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e31665640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6e2c0a9d70 0x7f6e2c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:55050/0 (socket says 172.21.15.94:55050) 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e31665640 1 -- 172.21.15.94:0/2084553370 learned_addr learned my addr 172.21.15.94:0/2084553370 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.532+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856668666 0 0) 0x7f6e2c0aaef0 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c003520 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235213334 0 0) 0x7f6e2c145eb0 con 0x7f6e2c1457f0 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:18.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c0aaef0 con 0x7f6e2c1457f0 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2097261471 0 0) 0x7f6e1c003520 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 >> v1:172.21.15.94:6790/0 conn(0x7f6e2c1457f0 legacy=0x7f6e2c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 >> v1:172.21.15.173:6789/0 conn(0x7f6e2c13f650 legacy=0x7f6e2c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 monclient: found mon.noname-a 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:41:18.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c146340 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e24001940 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: dump: 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:18.523 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e330c8640 5 monclient: authenticate success, global_id 4317 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e330c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e24001b60 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 1 -- 172.21.15.94:0/2084553370 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e24001ef0 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e2b7fe640 10 monclient: dump: 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.524 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e330c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.533+0000 7f6e330c8640 4 set_mon_vals no callback set 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.534+0000 7f6e330c8640 10 monclient: shutdown 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.534+0000 7f6e330c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.534+0000 7f6e330c8640 1 -- 172.21.15.94:0/2084553370 >> v1:172.21.15.94:6789/0 conn(0x7f6e2c0a9d70 legacy=0x7f6e2c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.534+0000 7f6e330c8640 1 -- 172.21.15.94:0/2084553370 shutdown_connections 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.534+0000 7f6e330c8640 1 -- 172.21.15.94:0/2084553370 >> 172.21.15.94:0/2084553370 conn(0x7f6e2c066a00 msgr2=0x7f6e2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.534+0000 7f6e330c8640 1 -- 172.21.15.94:0/2084553370 shutdown_connections 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.534+0000 7f6e330c8640 1 -- 172.21.15.94:0/2084553370 wait complete. 2024-03-29T10:41:18.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.535+0000 7f6e330c8640 10 monclient: build_initial_monmap 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.535+0000 7f6e330c8640 10 monclient: monmap: 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.536+0000 7f6e330c8640 1 librados: starting msgr at 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.536+0000 7f6e330c8640 1 librados: starting objecter 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.536+0000 7f6e330c8640 1 Processor -- start 2024-03-29T10:41:18.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.536+0000 7f6e330c8640 1 -- start start 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.536+0000 7f6e330c8640 1 librados: setting wanted keys 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.536+0000 7f6e330c8640 1 librados: calling monclient init 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.536+0000 7f6e330c8640 10 monclient: init 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c0a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c13f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c1457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c0ac500 con 0x7f6e2c1457f0 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c13c790 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c13c920 con 0x7f6e2c13f650 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:18.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e330c8640 10 monclient(hunting): authenticate will time out at 929.805829s 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e30e64640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f6e2c0a9d70 0x7f6e2c06ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:40978/0 (socket says 172.21.15.94:40978) 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e30e64640 1 -- 172.21.15.94:0/987700254 learned_addr learned my addr 172.21.15.94:0/987700254 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e297fa640 10 client.?.objecter ms_handle_connect 0x7f6e2c0a9d70 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.537+0000 7f6e297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968854321 0 0) 0x7f6e2c13c790 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e10004eb0 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 client.?.objecter ms_handle_connect 0x7f6e2c1457f0 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 client.?.objecter ms_handle_connect 0x7f6e2c13f650 2024-03-29T10:41:18.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3579270951 0 0) 0x7f6e10004eb0 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e2c13c790 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e20001970 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): dump: 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:18.529 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407571575 0 0) 0x7f6e2c0ac500 con 0x7f6e2c1457f0 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e10004eb0 con 0x7f6e2c1457f0 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.538+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1817069763 0 0) 0x7f6e2c13c790 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 >> v1:172.21.15.94:6790/0 conn(0x7f6e2c13f650 legacy=0x7f6e2c070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 >> v1:172.21.15.94:6789/0 conn(0x7f6e2c1457f0 legacy=0x7f6e2c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: _finish_hunting 0 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 1 monclient: found mon.b 2024-03-29T10:41:18.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c0b1670 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: _finish_auth 0 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: _check_auth_tickets 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 10 monclient: _renew_subs 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c13bcf0 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 10 monclient: _renew_subs 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c1e7580 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e330c8640 1 librados: init done 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e20003610 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e20003a20 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e297fa640 10 monclient: dump: 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:18.532 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:18.533 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:18.533 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:18.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.539+0000 7f6e2affd640 4 set_mon_vals no callback set 2024-03-29T10:41:18.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.541+0000 7f6e330c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:18.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.541+0000 7f6e330c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:18.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.541+0000 7f6e330c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:18.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.541+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c005080 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.541+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f6e20003b80 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.542+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f6e20084a40 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.542+0000 7f6e297fa640 10 client.4306.objecter ms_dispatch 0x7f6e2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.542+0000 7f6e297fa640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.542+0000 7f6e297fa640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.542+0000 7f6e2bfff640 10 monclient: get_auth_request con 0x7f6e1006ddc0 auth_method 0 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.542+0000 7f6e297fa640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.542+0000 7f6e297fa640 10 client.4306.objecter ms_handle_connect 0x7f6e1006ddc0 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.550+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e20050320 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.550+0000 7f6e297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:18.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.550+0000 7f6e297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:18.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.929+0000 7f6e330c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:18.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.929+0000 7f6e330c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:18.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.929+0000 7f6e330c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:18.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.929+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e0c005330 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.930+0000 7f6e297fa640 1 -- 172.21.15.94:0/987700254 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f6e20052250 con 0x7f6e2c0a9d70 2024-03-29T10:41:18.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.930+0000 7f6e297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:18.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.930+0000 7f6e297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:18.914 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:18.914 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.933+0000 7f6e330c8640 10 librados: watch_flush enter 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.933+0000 7f6e330c8640 10 librados: watch_flush exit 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.933+0000 7f6e330c8640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.933+0000 7f6e330c8640 10 client.4306.objecter successfully canceled tick 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 >> v1:172.21.15.94:6816/3829934808 conn(0x7f6e1006ddc0 legacy=0x7f6e10070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 10 monclient: shutdown 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 >> v1:172.21.15.173:6789/0 conn(0x7f6e2c0a9d70 legacy=0x7f6e2c06ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 shutdown_connections 2024-03-29T10:41:18.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 >> 172.21.15.94:0/987700254 conn(0x7f6e2c066a00 msgr2=0x7f6e2c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:18.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 shutdown_connections 2024-03-29T10:41:18.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 1 -- 172.21.15.94:0/987700254 wait complete. 2024-03-29T10:41:18.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:18.934+0000 7f6e330c8640 1 librados: shutdown 2024-03-29T10:41:18.933 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:25.740 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.893+0000 7f0bc881b640 10 monclient: get_monmap_and_config 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.893+0000 7f0bc881b640 10 monclient: build_initial_monmap 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.893+0000 7f0bc881b640 10 monclient: monmap: 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:25.894463+0000 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:25.894463+0000 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:25.877 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:25.878 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:25.878 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:25.878 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:25.878 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:25.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.894+0000 7f0bc881b640 1 Processor -- start 2024-03-29T10:41:25.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.894+0000 7f0bc881b640 1 -- start start 2024-03-29T10:41:25.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.894+0000 7f0bc881b640 10 monclient: init 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.895+0000 7f0bc881b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.895+0000 7f0bc881b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.895+0000 7f0bc881b640 10 monclient(hunting): picked mon.noname-c con 0x7f0bbc0389e0 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc37fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0bbc0389e0 0x7f0bbc033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:57830/0 (socket says 172.21.15.94:57830) 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc881b640 10 monclient(hunting): picked mon.noname-b con 0x7f0bbc038dc0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc37fe640 1 -- 172.21.15.94:0/2039563893 learned_addr learned my addr 172.21.15.94:0/2039563893 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc881b640 10 monclient(hunting): picked mon.noname-a con 0x7f0bbc030660 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0c54c0 con 0x7f0bbc030660 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc036430 con 0x7f0bbc0389e0 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc039840 con 0x7f0bbc038dc0 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc881b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:25.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.896+0000 7f0bc881b640 10 monclient(hunting): authenticate will time out at 937.164423s 2024-03-29T10:41:25.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622787509 0 0) 0x7f0bbc036430 con 0x7f0bbc0389e0 2024-03-29T10:41:25.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:25.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:25.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:25.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb4003520 con 0x7f0bbc0389e0 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818714422 0 0) 0x7f0bbc0c54c0 con 0x7f0bbc030660 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc036430 con 0x7f0bbc030660 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2996275277 0 0) 0x7f0bb4003520 con 0x7f0bbc0389e0 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.897+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 >> v1:172.21.15.94:6790/0 conn(0x7f0bbc038dc0 legacy=0x7f0bbc0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 >> v1:172.21.15.94:6789/0 conn(0x7f0bbc030660 legacy=0x7f0bbc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 1 monclient: found mon.noname-c 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:41:25.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc035690 con 0x7f0bbc0389e0 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0bb8001690 con 0x7f0bbc0389e0 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: dump: 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:25.885 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc881b640 5 monclient: authenticate success, global_id 4309 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc881b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bb80018f0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 1 -- 172.21.15.94:0/2039563893 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0bb8001d00 con 0x7f0bbc0389e0 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.899+0000 7f0bc27fc640 10 monclient: dump: 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:25.886 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.900+0000 7f0bc881b640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:25.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.900+0000 7f0bc881b640 4 set_mon_vals no callback set 2024-03-29T10:41:25.892 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.900+0000 7f0bc881b640 10 monclient: shutdown 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.901+0000 7f0bc881b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.901+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 >> v1:172.21.15.173:6789/0 conn(0x7f0bbc0389e0 legacy=0x7f0bbc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.901+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 shutdown_connections 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.901+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 >> 172.21.15.94:0/2039563893 conn(0x7f0bbc005c00 msgr2=0x7f0bbc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.901+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 shutdown_connections 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.901+0000 7f0bc881b640 1 -- 172.21.15.94:0/2039563893 wait complete. 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.902+0000 7f0bc881b640 10 monclient: build_initial_monmap 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.902+0000 7f0bc881b640 10 monclient: monmap: 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:25.893 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.902+0000 7f0bc881b640 1 librados: starting msgr at 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.902+0000 7f0bc881b640 1 librados: starting objecter 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.902+0000 7f0bc881b640 1 Processor -- start 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.903+0000 7f0bc881b640 1 -- start start 2024-03-29T10:41:25.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.903+0000 7f0bc881b640 1 librados: setting wanted keys 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.903+0000 7f0bc881b640 1 librados: calling monclient init 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.903+0000 7f0bc881b640 10 monclient: init 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 10 monclient(hunting): picked mon.noname-b con 0x7f0bbc030660 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 10 monclient(hunting): picked mon.noname-a con 0x7f0bbc0389e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 10 monclient(hunting): picked mon.noname-c con 0x7f0bbc0c73a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0a52c0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc02eed0 con 0x7f0bbc030660 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc02ea00 con 0x7f0bbc0c73a0 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:25.895 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc881b640 10 monclient(hunting): authenticate will time out at 937.172527s 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc37fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0bbc030660 0x7f0bbc15f9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:58814/0 (socket says 172.21.15.94:58814) 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.904+0000 7f0bc37fe640 1 -- 172.21.15.94:0/517125471 learned_addr learned my addr 172.21.15.94:0/517125471 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 client.?.objecter ms_handle_connect 0x7f0bbc030660 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 client.?.objecter ms_handle_connect 0x7f0bbc0389e0 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469139447 0 0) 0x7f0bbc02eed0 con 0x7f0bbc030660 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba8004eb0 con 0x7f0bbc030660 2024-03-29T10:41:25.896 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115901563 0 0) 0x7f0bbc0a52c0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc02eed0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.905+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 795080338 0 0) 0x7f0bbc02eed0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bbc0a52c0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0bb0001640 con 0x7f0bbc0389e0 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 10 monclient(hunting): dump: 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.897 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1066385355 0 0) 0x7f0bbc0a52c0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.906+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 >> v1:172.21.15.94:6790/0 conn(0x7f0bbc0c73a0 legacy=0x7f0bbc0c1f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 >> v1:172.21.15.173:6789/0 conn(0x7f0bbc030660 legacy=0x7f0bbc15f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 1 monclient: found mon.a 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:25.898 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc0c2630 con 0x7f0bbc0389e0 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: _finish_auth 0 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 10 monclient: _renew_subs 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc02f3f0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bb00017a0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0bb00035f0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:25.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0baffff640 10 monclient: dump: 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc1ffb640 4 set_mon_vals no callback set 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 10 monclient: _renew_subs 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:25.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc168d70 con 0x7f0bbc0389e0 2024-03-29T10:41:25.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.907+0000 7f0bc881b640 1 librados: init done 2024-03-29T10:41:25.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.909+0000 7f0bc881b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:25.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.909+0000 7f0bc881b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:25.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.909+0000 7f0bc881b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:25.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.909+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc007b90 con 0x7f0bbc0389e0 2024-03-29T10:41:25.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.909+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f0bb00195c0 con 0x7f0bbc0389e0 2024-03-29T10:41:25.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.910+0000 7f0bc37fe640 10 monclient: get_auth_request con 0x7f0ba806db20 auth_method 0 2024-03-29T10:41:25.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.911+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0bb0083620 con 0x7f0bbc0389e0 2024-03-29T10:41:25.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.911+0000 7f0baffff640 10 client.4326.objecter ms_dispatch 0x7f0bc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:25.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.911+0000 7f0baffff640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:25.903 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.911+0000 7f0baffff640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:25.903 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.911+0000 7f0baffff640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T10:41:25.903 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.911+0000 7f0baffff640 10 client.4326.objecter ms_handle_connect 0x7f0ba806db20 2024-03-29T10:41:25.903 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.916+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bb004ed30 con 0x7f0bbc0389e0 2024-03-29T10:41:25.903 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.916+0000 7f0baffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:25.903 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:25.916+0000 7f0baffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:26.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.235+0000 7f0bc881b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:26.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.235+0000 7f0bc881b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:26.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.235+0000 7f0bc881b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:26.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.235+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bbc02ed20 con 0x7f0bbc0389e0 2024-03-29T10:41:26.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.237+0000 7f0baffff640 1 -- 172.21.15.94:0/517125471 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f0bb0050e30 con 0x7f0bbc0389e0 2024-03-29T10:41:26.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.237+0000 7f0baffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:26.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.237+0000 7f0baffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:26.220 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:26.220 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:26.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.240+0000 7f0bc881b640 10 librados: watch_flush enter 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.240+0000 7f0bc881b640 10 librados: watch_flush exit 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.240+0000 7f0bc881b640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.240+0000 7f0bc881b640 10 client.4326.objecter successfully canceled tick 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.241+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 >> v1:172.21.15.94:6816/3829934808 conn(0x7f0ba806db20 legacy=0x7f0ba806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.241+0000 7f0bc881b640 10 monclient: shutdown 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.241+0000 7f0bc881b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.241+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 >> v1:172.21.15.94:6789/0 conn(0x7f0bbc0389e0 legacy=0x7f0bbc02e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:26.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.241+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 shutdown_connections 2024-03-29T10:41:26.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.241+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 >> 172.21.15.94:0/517125471 conn(0x7f0bbc005c00 msgr2=0x7f0bbc034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:26.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.243+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 shutdown_connections 2024-03-29T10:41:26.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.243+0000 7f0bc881b640 1 -- 172.21.15.94:0/517125471 wait complete. 2024-03-29T10:41:26.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:26.244+0000 7f0bc881b640 1 librados: shutdown 2024-03-29T10:41:26.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:33.054 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.229+0000 7fcabbf90640 10 monclient: get_monmap_and_config 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.229+0000 7fcabbf90640 10 monclient: build_initial_monmap 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.229+0000 7fcabbf90640 10 monclient: monmap: 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:33.230688+0000 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:33.230688+0000 2024-03-29T10:41:33.214 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:33.215 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:33.215 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:33.215 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:33.215 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:33.215 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:33.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.232+0000 7fcabbf90640 1 Processor -- start 2024-03-29T10:41:33.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.232+0000 7fcabbf90640 1 -- start start 2024-03-29T10:41:33.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.232+0000 7fcabbf90640 10 monclient: init 2024-03-29T10:41:33.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.233+0000 7fcabbf90640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:33.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.233+0000 7fcabbf90640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:33.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.236+0000 7fcabbf90640 10 monclient(hunting): picked mon.noname-c con 0x7fcab40a9ac0 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:33.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.237+0000 7fcabbf90640 10 monclient(hunting): picked mon.noname-b con 0x7fcab413ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:33.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.239+0000 7fcab9d2c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fcab40a9ac0 0x7fcab40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:51845/0 (socket says 172.21.15.94:51845) 2024-03-29T10:41:33.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcaba52d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fcab40af900 0x7fcab4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36973/0 (socket says 172.21.15.94:36973) 2024-03-29T10:41:33.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcab9d2c640 1 -- 172.21.15.94:0/2849402227 learned_addr learned my addr 172.21.15.94:0/2849402227 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:33.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcabbf90640 10 monclient(hunting): picked mon.noname-a con 0x7fcab40af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:33.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40afd10 con 0x7fcab40af900 2024-03-29T10:41:33.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40af6f0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40aac80 con 0x7fcab413ee90 2024-03-29T10:41:33.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcabbf90640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:33.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.240+0000 7fcabbf90640 10 monclient(hunting): authenticate will time out at 944.508551s 2024-03-29T10:41:33.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.243+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490915405 0 0) 0x7fcab40afd10 con 0x7fcab40af900 2024-03-29T10:41:33.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.243+0000 7fcab8d2a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:33.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.243+0000 7fcab8d2a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:33.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.243+0000 7fcab8d2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:33.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.243+0000 7fcab8d2a640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:41:33.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.243+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca9c003440 con 0x7fcab40af900 2024-03-29T10:41:33.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.244+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4262823137 0 0) 0x7fca9c003440 con 0x7fcab40af900 2024-03-29T10:41:33.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.244+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 >> v1:172.21.15.94:6790/0 conn(0x7fcab413ee90 legacy=0x7fcab413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:33.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.244+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 >> v1:172.21.15.173:6789/0 conn(0x7fcab40a9ac0 legacy=0x7fcab40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 10 monclient: _finish_hunting 0 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 1 monclient: found mon.noname-a 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40aaf60 con 0x7fcab40af900 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 10 monclient: _finish_auth 0 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 10 monclient: _check_auth_tickets 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:33.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcaa8004c00 con 0x7fcab40af900 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.245+0000 7fcab8d2a640 10 monclient: dump: 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:33.231 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:33.232 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:33.232 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:33.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.247+0000 7fcabbf90640 5 monclient: authenticate success, global_id 4329 2024-03-29T10:41:33.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.247+0000 7fcabbf90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:33.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaa8004d60 con 0x7fcab40af900 2024-03-29T10:41:33.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcab8d2a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:33.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcab8d2a640 1 -- 172.21.15.94:0/2849402227 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcaa80013f0 con 0x7fcab40af900 2024-03-29T10:41:33.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcab8d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:33.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcab8d2a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:33.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcab8d2a640 10 monclient: dump: 2024-03-29T10:41:33.233 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:33.233 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcabbf90640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.248+0000 7fcabbf90640 4 set_mon_vals no callback set 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.249+0000 7fcabbf90640 10 monclient: shutdown 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.249+0000 7fcabbf90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:33.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.249+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 >> v1:172.21.15.94:6789/0 conn(0x7fcab40af900 legacy=0x7fcab4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:33.235 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.249+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 shutdown_connections 2024-03-29T10:41:33.235 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.249+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 >> 172.21.15.94:0/2849402227 conn(0x7fcab40b35e0 msgr2=0x7fcab40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:33.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.251+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 shutdown_connections 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.251+0000 7fcabbf90640 1 -- 172.21.15.94:0/2849402227 wait complete. 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 10 monclient: build_initial_monmap 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 10 monclient: monmap: 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:33.237 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 1 librados: starting msgr at 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 1 librados: starting objecter 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 1 Processor -- start 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 1 -- start start 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 1 librados: setting wanted keys 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 1 librados: calling monclient init 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.252+0000 7fcabbf90640 10 monclient: init 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 10 monclient(hunting): picked mon.noname-c con 0x7fcab40a9ac0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 10 monclient(hunting): picked mon.noname-b con 0x7fcab40af900 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 10 monclient(hunting): picked mon.noname-a con 0x7fcab413ee90 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab41c75c0 con 0x7fcab413ee90 2024-03-29T10:41:33.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab4072480 con 0x7fcab40af900 2024-03-29T10:41:33.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab41f9c30 con 0x7fcab40a9ac0 2024-03-29T10:41:33.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:33.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.253+0000 7fcabbf90640 10 monclient(hunting): authenticate will time out at 944.521927s 2024-03-29T10:41:33.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.257+0000 7fcab9d2c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fcab40a9ac0 0x7fcab4070270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:42149/0 (socket says 172.21.15.94:42149) 2024-03-29T10:41:33.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.257+0000 7fcab9d2c640 1 -- 172.21.15.94:0/3234846859 learned_addr learned my addr 172.21.15.94:0/3234846859 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:33.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.257+0000 7fcab27fc640 10 client.?.objecter ms_handle_connect 0x7fcab40a9ac0 2024-03-29T10:41:33.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.257+0000 7fcab27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.257+0000 7fcab27fc640 10 client.?.objecter ms_handle_connect 0x7fcab413ee90 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.257+0000 7fcab27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528706422 0 0) 0x7fcab41c75c0 con 0x7fcab413ee90 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:41:33.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaa0004eb0 con 0x7fcab413ee90 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212558737 0 0) 0x7fcab41f9c30 con 0x7fcab40a9ac0 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab41c75c0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080520813 0 0) 0x7fcaa0004eb0 con 0x7fcab413ee90 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa0005490 con 0x7fcab413ee90 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcaa80013b0 con 0x7fcab413ee90 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:33.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): dump: 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:33.244 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.258+0000 7fcab27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222686766 0 0) 0x7fcab41c75c0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaa0004eb0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcaa4001c70 con 0x7fcab40a9ac0 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 10 monclient(hunting): dump: 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:33.245 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:33.246 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:33.246 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:33.246 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 146012341 0 0) 0x7fcaa0004eb0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.246 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.259+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 >> v1:172.21.15.173:6789/0 conn(0x7fcab40af900 legacy=0x7fcab41f8e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:33.249 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 >> v1:172.21.15.94:6789/0 conn(0x7fcab413ee90 legacy=0x7fcab41f9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:33.249 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:33.249 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 1 monclient: found mon.c 2024-03-29T10:41:33.249 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab4071bc0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 10 monclient: _renew_subs 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:33.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab4071ea0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 10 monclient: _renew_subs 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaa4004b60 con 0x7fcab40a9ac0 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab406fa40 con 0x7fcab40a9ac0 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcabbf90640 1 librados: init done 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.260+0000 7fcab37fe640 4 set_mon_vals no callback set 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.261+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcaa4001c70 con 0x7fcab40a9ac0 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.261+0000 7fcab27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.261+0000 7fcab27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.261+0000 7fcab27fc640 10 monclient: dump: 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:33.251 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.262+0000 7fcabbf90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.262+0000 7fcabbf90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:33.252 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.263+0000 7fcabbf90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.263+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fcaa401abc0 con 0x7fcab40a9ac0 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.263+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab40aa900 con 0x7fcab40a9ac0 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.265+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fcaa4084980 con 0x7fcab40a9ac0 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.265+0000 7fcab27fc640 10 client.4328.objecter ms_dispatch 0x7fcab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.265+0000 7fcab27fc640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.265+0000 7fcab27fc640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.265+0000 7fcab27fc640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T10:41:33.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.265+0000 7fcab952b640 10 monclient: get_auth_request con 0x7fcaa006e330 auth_method 0 2024-03-29T10:41:33.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.267+0000 7fcab27fc640 10 client.4328.objecter ms_handle_connect 0x7fcaa006e330 2024-03-29T10:41:33.260 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.277+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcaa4050260 con 0x7fcab40a9ac0 2024-03-29T10:41:33.260 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.277+0000 7fcab27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:33.260 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.277+0000 7fcab27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:33.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.597+0000 7fcabbf90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:33.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.597+0000 7fcabbf90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:33.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.597+0000 7fcabbf90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:41:33.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.597+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcab4075f10 con 0x7fcab40a9ac0 2024-03-29T10:41:33.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.598+0000 7fcab27fc640 1 -- 172.21.15.94:0/3234846859 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fcaa4052190 con 0x7fcab40a9ac0 2024-03-29T10:41:33.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.598+0000 7fcab27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:33.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.598+0000 7fcab27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:33.585 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:33.585 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:33.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.604+0000 7fcabbf90640 10 librados: watch_flush enter 2024-03-29T10:41:33.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.604+0000 7fcabbf90640 10 librados: watch_flush exit 2024-03-29T10:41:33.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.604+0000 7fcabbf90640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T10:41:33.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.604+0000 7fcabbf90640 10 client.4328.objecter successfully canceled tick 2024-03-29T10:41:33.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.605+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 >> v1:172.21.15.94:6816/3829934808 conn(0x7fcaa006e330 legacy=0x7fcaa00707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:33.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.605+0000 7fcabbf90640 10 monclient: shutdown 2024-03-29T10:41:33.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.605+0000 7fcabbf90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:33.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.605+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 >> v1:172.21.15.94:6790/0 conn(0x7fcab40a9ac0 legacy=0x7fcab4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:33.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.607+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 shutdown_connections 2024-03-29T10:41:33.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.607+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 >> 172.21.15.94:0/3234846859 conn(0x7fcab40b35e0 msgr2=0x7fcab41d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:33.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.610+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 shutdown_connections 2024-03-29T10:41:33.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.610+0000 7fcabbf90640 1 -- 172.21.15.94:0/3234846859 wait complete. 2024-03-29T10:41:33.594 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:33.611+0000 7fcabbf90640 1 librados: shutdown 2024-03-29T10:41:33.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:40.421 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:40.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.597+0000 7fe705205640 10 monclient: get_monmap_and_config 2024-03-29T10:41:40.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.597+0000 7fe705205640 10 monclient: build_initial_monmap 2024-03-29T10:41:40.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.597+0000 7fe705205640 10 monclient: monmap: 2024-03-29T10:41:40.582 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:40.582 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:40.582 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:40.599440+0000 2024-03-29T10:41:40.582 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:40.599440+0000 2024-03-29T10:41:40.583 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:40.583 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:40.583 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:40.583 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:40.583 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:40.583 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:40.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.600+0000 7fe705205640 1 Processor -- start 2024-03-29T10:41:40.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.600+0000 7fe705205640 1 -- start start 2024-03-29T10:41:40.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.600+0000 7fe705205640 10 monclient: init 2024-03-29T10:41:40.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:40.586 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 10 monclient(hunting): picked mon.noname-b con 0x7fe7001457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 10 monclient(hunting): picked mon.noname-a con 0x7fe70013f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 10 monclient(hunting): picked mon.noname-c con 0x7fe7000a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7000aaef0 con 0x7fe70013f650 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe700145bd0 con 0x7fe7000a9d70 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe700145eb0 con 0x7fe7001457f0 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:40.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.601+0000 7fe705205640 10 monclient(hunting): authenticate will time out at 951.870638s 2024-03-29T10:41:40.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.605+0000 7fe6fed9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe7001457f0 0x7fe70013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:42819/0 (socket says 172.21.15.94:42819) 2024-03-29T10:41:40.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.605+0000 7fe6fed9d640 1 -- 172.21.15.94:0/1954456443 learned_addr learned my addr 172.21.15.94:0/1954456443 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:40.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.606+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425518022 0 0) 0x7fe7000aaef0 con 0x7fe70013f650 2024-03-29T10:41:40.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.606+0000 7fe6fdd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:40.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.606+0000 7fe6fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:40.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.606+0000 7fe6fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:40.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.606+0000 7fe6fdd9b640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:41:40.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.606+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e8003520 con 0x7fe70013f650 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698972752 0 0) 0x7fe6e8003520 con 0x7fe70013f650 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 >> v1:172.21.15.94:6790/0 conn(0x7fe7001457f0 legacy=0x7fe70013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 >> v1:172.21.15.173:6789/0 conn(0x7fe7000a9d70 legacy=0x7fe70014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 1 monclient: found mon.noname-a 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe700146340 con 0x7fe70013f650 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 10 monclient: _finish_auth 0 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe6f0001690 con 0x7fe70013f650 2024-03-29T10:41:40.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe6fdd9b640 10 monclient: dump: 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:40.597 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe705205640 5 monclient: authenticate success, global_id 4335 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.609+0000 7fe705205640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6f00017f0 con 0x7fe70013f650 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe6fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe6fdd9b640 1 -- 172.21.15.94:0/1954456443 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe6f0001c40 con 0x7fe70013f650 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe6fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe6fdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe6fdd9b640 10 monclient: dump: 2024-03-29T10:41:40.598 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:40.599 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:40.599 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe705205640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:40.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.611+0000 7fe705205640 4 set_mon_vals no callback set 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.612+0000 7fe705205640 10 monclient: shutdown 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.612+0000 7fe705205640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.612+0000 7fe705205640 1 -- 172.21.15.94:0/1954456443 >> v1:172.21.15.94:6789/0 conn(0x7fe70013f650 legacy=0x7fe7000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.612+0000 7fe705205640 1 -- 172.21.15.94:0/1954456443 shutdown_connections 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.612+0000 7fe705205640 1 -- 172.21.15.94:0/1954456443 >> 172.21.15.94:0/1954456443 conn(0x7fe700066a00 msgr2=0x7fe700066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.612+0000 7fe705205640 1 -- 172.21.15.94:0/1954456443 shutdown_connections 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.612+0000 7fe705205640 1 -- 172.21.15.94:0/1954456443 wait complete. 2024-03-29T10:41:40.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.613+0000 7fe705205640 10 monclient: build_initial_monmap 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.613+0000 7fe705205640 10 monclient: monmap: 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.614+0000 7fe705205640 1 librados: starting msgr at 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.614+0000 7fe705205640 1 librados: starting objecter 2024-03-29T10:41:40.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.614+0000 7fe705205640 1 Processor -- start 2024-03-29T10:41:40.604 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.614+0000 7fe705205640 1 -- start start 2024-03-29T10:41:40.604 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.614+0000 7fe705205640 1 librados: setting wanted keys 2024-03-29T10:41:40.604 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.614+0000 7fe705205640 1 librados: calling monclient init 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.614+0000 7fe705205640 10 monclient: init 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 10 monclient(hunting): picked mon.noname-c con 0x7fe7000a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 10 monclient(hunting): picked mon.noname-b con 0x7fe70013f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 10 monclient(hunting): picked mon.noname-a con 0x7fe7001457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7001337b0 con 0x7fe7001457f0 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70006f270 con 0x7fe70013f650 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7000ac460 con 0x7fe7000a9d70 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:40.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.615+0000 7fe705205640 10 monclient(hunting): authenticate will time out at 951.884756s 2024-03-29T10:41:40.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.621+0000 7fe6fe59c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe70013f650 0x7fe70006e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45189/0 (socket says 172.21.15.94:45189) 2024-03-29T10:41:40.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.621+0000 7fe6fe59c640 1 -- 172.21.15.94:0/2233246291 learned_addr learned my addr 172.21.15.94:0/2233246291 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.622+0000 7fe6e77fe640 10 client.?.objecter ms_handle_connect 0x7fe70013f650 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.622+0000 7fe6e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.622+0000 7fe6e77fe640 10 client.?.objecter ms_handle_connect 0x7fe7001457f0 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.622+0000 7fe6e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667531615 0 0) 0x7fe70006f270 con 0x7fe70013f650 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ec004eb0 con 0x7fe70013f650 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574299658 0 0) 0x7fe7001337b0 con 0x7fe7001457f0 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:40.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:40.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:41:40.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.623+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ec0051d0 con 0x7fe7001457f0 2024-03-29T10:41:40.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.624+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2423046519 0 0) 0x7fe6ec004eb0 con 0x7fe70013f650 2024-03-29T10:41:40.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.624+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6ec005640 con 0x7fe70013f650 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.624+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe6f0001c00 con 0x7fe70013f650 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.624+0000 7fe6e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.624+0000 7fe6e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.624+0000 7fe6e77fe640 10 monclient(hunting): dump: 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:40.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.624+0000 7fe6e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 10 client.?.objecter ms_handle_connect 0x7fe7000a9d70 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339117991 0 0) 0x7fe7000ac460 con 0x7fe7000a9d70 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.625+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ec005be0 con 0x7fe7000a9d70 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2959817649 0 0) 0x7fe6ec005640 con 0x7fe70013f650 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 >> v1:172.21.15.94:6790/0 conn(0x7fe7000a9d70 legacy=0x7fe70006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 >> v1:172.21.15.94:6789/0 conn(0x7fe7001457f0 legacy=0x7fe7001e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 1 monclient: found mon.b 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:40.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70006ef20 con 0x7fe70013f650 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe6e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 5 monclient: authenticate success, global_id 4318 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 10 monclient: _renew_subs 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70006f750 con 0x7fe70013f650 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 10 monclient: _renew_subs 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70013bf10 con 0x7fe70013f650 2024-03-29T10:41:40.613 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.626+0000 7fe705205640 1 librados: init done 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.629+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6f0003960 con 0x7fe70013f650 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.629+0000 7fe6e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.629+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe6f0001c00 con 0x7fe70013f650 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.629+0000 7fe6e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.629+0000 7fe6e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.629+0000 7fe6e77fe640 10 monclient: dump: 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:40.614 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:40.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.629+0000 7fe6fd59a640 4 set_mon_vals no callback set 2024-03-29T10:41:40.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.631+0000 7fe705205640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:40.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.631+0000 7fe705205640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:40.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.631+0000 7fe705205640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.631+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e0005080 con 0x7fe70013f650 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.631+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fe6f0003c90 con 0x7fe70013f650 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.632+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fe6f00545c0 con 0x7fe70013f650 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.632+0000 7fe6e77fe640 10 client.4318.objecter ms_dispatch 0x7fe700000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.632+0000 7fe6e77fe640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.632+0000 7fe6e77fe640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.632+0000 7fe6e77fe640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.632+0000 7fe6fed9d640 10 monclient: get_auth_request con 0x7fe6ec06e5e0 auth_method 0 2024-03-29T10:41:40.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.633+0000 7fe6e77fe640 10 client.4318.objecter ms_handle_connect 0x7fe6ec06e5e0 2024-03-29T10:41:40.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.641+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6f0050860 con 0x7fe70013f650 2024-03-29T10:41:40.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.641+0000 7fe6e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:40.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.641+0000 7fe6e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:40.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.959+0000 7fe705205640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:40.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.959+0000 7fe705205640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:40.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.959+0000 7fe705205640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:40.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.959+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6e0005330 con 0x7fe70013f650 2024-03-29T10:41:40.949 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.965+0000 7fe6e77fe640 1 -- 172.21.15.94:0/2233246291 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fe6f0050860 con 0x7fe70013f650 2024-03-29T10:41:40.949 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.965+0000 7fe6e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:40.949 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.965+0000 7fe6e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:40.950 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:40.950 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:40.952 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.968+0000 7fe705205640 10 librados: watch_flush enter 2024-03-29T10:41:40.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.969+0000 7fe705205640 10 librados: watch_flush exit 2024-03-29T10:41:40.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.969+0000 7fe705205640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T10:41:40.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.969+0000 7fe705205640 10 client.4318.objecter successfully canceled tick 2024-03-29T10:41:40.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.969+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 >> v1:172.21.15.94:6816/3829934808 conn(0x7fe6ec06e5e0 legacy=0x7fe6ec070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:40.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.969+0000 7fe705205640 10 monclient: shutdown 2024-03-29T10:41:40.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.969+0000 7fe705205640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:40.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.969+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 >> v1:172.21.15.173:6789/0 conn(0x7fe70013f650 legacy=0x7fe70006e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:40.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.972+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 shutdown_connections 2024-03-29T10:41:40.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.972+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 >> 172.21.15.94:0/2233246291 conn(0x7fe700066a00 msgr2=0x7fe70013d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:40.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.974+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 shutdown_connections 2024-03-29T10:41:40.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.976+0000 7fe705205640 1 -- 172.21.15.94:0/2233246291 wait complete. 2024-03-29T10:41:40.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:40.977+0000 7fe705205640 1 librados: shutdown 2024-03-29T10:41:40.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:47.786 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.956+0000 7fd8ed8d9640 10 monclient: get_monmap_and_config 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.956+0000 7fd8ed8d9640 10 monclient: build_initial_monmap 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.956+0000 7fd8ed8d9640 10 monclient: monmap: 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:47.957492+0000 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:47.957492+0000 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:47.940 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:47.941 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:47.941 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:47.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.958+0000 7fd8ed8d9640 1 Processor -- start 2024-03-29T10:41:47.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.959+0000 7fd8ed8d9640 1 -- start start 2024-03-29T10:41:47.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.959+0000 7fd8ed8d9640 10 monclient: init 2024-03-29T10:41:47.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.960+0000 7fd8ed8d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:47.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.960+0000 7fd8ed8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:47.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.960+0000 7fd8ed8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fd8e81457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.964+0000 7fd8e6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd8e81457f0 0x7fd8e813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59291/0 (socket says 172.21.15.94:59291) 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.965+0000 7fd8ed8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fd8e813f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.965+0000 7fd8e6ffd640 1 -- 172.21.15.94:0/581899941 learned_addr learned my addr 172.21.15.94:0/581899941 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.967+0000 7fd8ed8d9640 10 monclient(hunting): picked mon.noname-b con 0x7fd8e80a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.967+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e80aaef0 con 0x7fd8e81457f0 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.967+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e8145bd0 con 0x7fd8e813f650 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.967+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e8145eb0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.967+0000 7fd8ed8d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:47.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.967+0000 7fd8ed8d9640 10 monclient(hunting): authenticate will time out at 959.235468s 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574131147 0 0) 0x7fd8e8145eb0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d0003520 con 0x7fd8e80a9d70 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3829689205 0 0) 0x7fd8d0003520 con 0x7fd8e80a9d70 2024-03-29T10:41:47.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 >> v1:172.21.15.173:6789/0 conn(0x7fd8e813f650 legacy=0x7fd8e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 >> v1:172.21.15.94:6789/0 conn(0x7fd8e81457f0 legacy=0x7fd8e813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8146340 con 0x7fd8e80a9d70 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd8e00019f0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: dump: 2024-03-29T10:41:47.959 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8e0001bd0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 1 -- 172.21.15.94:0/581899941 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd8e00019f0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:47.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.970+0000 7fd8e5ffb640 10 monclient: dump: 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.971+0000 7fd8ed8d9640 5 monclient: authenticate success, global_id 4334 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.971+0000 7fd8ed8d9640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.971+0000 7fd8ed8d9640 4 set_mon_vals no callback set 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.972+0000 7fd8ed8d9640 10 monclient: shutdown 2024-03-29T10:41:47.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.972+0000 7fd8ed8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.972+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 >> v1:172.21.15.94:6790/0 conn(0x7fd8e80a9d70 legacy=0x7fd8e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.972+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 shutdown_connections 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.972+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 >> 172.21.15.94:0/581899941 conn(0x7fd8e8066a00 msgr2=0x7fd8e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.972+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 shutdown_connections 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.972+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/581899941 wait complete. 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.973+0000 7fd8ed8d9640 10 monclient: build_initial_monmap 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.973+0000 7fd8ed8d9640 10 monclient: monmap: 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:47.962 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.974+0000 7fd8ed8d9640 1 librados: starting msgr at 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.974+0000 7fd8ed8d9640 1 librados: starting objecter 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.974+0000 7fd8ed8d9640 1 Processor -- start 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.975+0000 7fd8ed8d9640 1 -- start start 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.975+0000 7fd8ed8d9640 1 librados: setting wanted keys 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.975+0000 7fd8ed8d9640 1 librados: calling monclient init 2024-03-29T10:41:47.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.975+0000 7fd8ed8d9640 10 monclient: init 2024-03-29T10:41:47.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.976+0000 7fd8ed8d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.976+0000 7fd8ed8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.976+0000 7fd8ed8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fd8e80a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.979+0000 7fd8ed8d9640 10 monclient(hunting): picked mon.noname-b con 0x7fd8e81457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.980+0000 7fd8e6ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fd8e80a9d70 0x7fd8e8139cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59399/0 (socket says 172.21.15.94:59399) 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.981+0000 7fd8ed8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fd8e8140ef0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.981+0000 7fd8ed8d9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e813b0c0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.981+0000 7fd8ed8d9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e813b3a0 con 0x7fd8e81457f0 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.981+0000 7fd8ed8d9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e813a3d0 con 0x7fd8e8140ef0 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.981+0000 7fd8ed8d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.981+0000 7fd8ed8d9640 10 monclient(hunting): authenticate will time out at 959.249196s 2024-03-29T10:41:47.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.981+0000 7fd8e6ffd640 1 -- 172.21.15.94:0/1345504207 learned_addr learned my addr 172.21.15.94:0/1345504207 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:47.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 10 client.?.objecter ms_handle_connect 0x7fd8e80a9d70 2024-03-29T10:41:47.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:47.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300000974 0 0) 0x7fd8e813b0c0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:47.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.984+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d4004eb0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 10 client.?.objecter ms_handle_connect 0x7fd8e81457f0 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519801477 0 0) 0x7fd8e813b3a0 con 0x7fd8e81457f0 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:41:47.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.985+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e813b0c0 con 0x7fd8e81457f0 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515181230 0 0) 0x7fd8d4004eb0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8e813b3a0 con 0x7fd8e80a9d70 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3407551910 0 0) 0x7fd8e813b0c0 con 0x7fd8e81457f0 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8d40058a0 con 0x7fd8e81457f0 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd8dc001820 con 0x7fd8e81457f0 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient(hunting): dump: 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:47.972 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 client.?.objecter ms_handle_connect 0x7fd8e8140ef0 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd8d8001810 con 0x7fd8e80a9d70 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient(hunting): dump: 2024-03-29T10:41:47.973 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3765043151 0 0) 0x7fd8d40058a0 con 0x7fd8e81457f0 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 >> v1:172.21.15.94:6790/0 conn(0x7fd8e8140ef0 legacy=0x7fd8e81e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 >> v1:172.21.15.94:6789/0 conn(0x7fd8e80a9d70 legacy=0x7fd8e8139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:47.974 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 monclient: found mon.b 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8141380 con 0x7fd8e81457f0 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient: _finish_auth 0 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 5 monclient: authenticate success, global_id 4324 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 10 monclient: _renew_subs 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e813bb10 con 0x7fd8e81457f0 2024-03-29T10:41:47.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 10 monclient: _renew_subs 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e81e3740 con 0x7fd8e81457f0 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.986+0000 7fd8ed8d9640 1 librados: init done 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cdffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cdffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c8005080 con 0x7fd8e81457f0 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8dc003610 con 0x7fd8e81457f0 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd8dc0039e0 con 0x7fd8e81457f0 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:47.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8cffff640 10 monclient: dump: 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:47.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.988+0000 7fd8e57fa640 4 set_mon_vals no callback set 2024-03-29T10:41:47.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.989+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fd8dc003c00 con 0x7fd8e81457f0 2024-03-29T10:41:47.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.990+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fd8dc083750 con 0x7fd8e81457f0 2024-03-29T10:41:47.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.990+0000 7fd8cffff640 10 client.4324.objecter ms_dispatch 0x7fd8e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:47.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.990+0000 7fd8cffff640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:47.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.990+0000 7fd8cffff640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:47.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.990+0000 7fd8cffff640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T10:41:47.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.990+0000 7fd8e6ffd640 10 monclient: get_auth_request con 0x7fd8d406e000 auth_method 0 2024-03-29T10:41:47.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.991+0000 7fd8cffff640 10 client.4324.objecter ms_handle_connect 0x7fd8d406e000 2024-03-29T10:41:47.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.998+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8dc04f030 con 0x7fd8e81457f0 2024-03-29T10:41:47.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.998+0000 7fd8cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:47.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:47.998+0000 7fd8cffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:48.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.335+0000 7fd8cdffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:48.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.335+0000 7fd8cdffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:48.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.335+0000 7fd8cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:41:48.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.335+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8c8005330 con 0x7fd8e81457f0 2024-03-29T10:41:48.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.336+0000 7fd8cffff640 1 -- 172.21.15.94:0/1345504207 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fd8dc050f60 con 0x7fd8e81457f0 2024-03-29T10:41:48.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.336+0000 7fd8cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:48.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.336+0000 7fd8cffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:48.322 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:48.322 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:48.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.342+0000 7fd8cdffb640 10 librados: watch_flush enter 2024-03-29T10:41:48.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.342+0000 7fd8cdffb640 10 librados: watch_flush exit 2024-03-29T10:41:48.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.342+0000 7fd8cdffb640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T10:41:48.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.342+0000 7fd8cdffb640 10 client.4324.objecter successfully canceled tick 2024-03-29T10:41:48.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.344+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 >> v1:172.21.15.94:6816/3829934808 conn(0x7fd8d406e000 legacy=0x7fd8d40704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:48.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.344+0000 7fd8cdffb640 10 monclient: shutdown 2024-03-29T10:41:48.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.344+0000 7fd8cdffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:48.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.344+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 >> v1:172.21.15.173:6789/0 conn(0x7fd8e81457f0 legacy=0x7fd8e81407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:48.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.345+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 shutdown_connections 2024-03-29T10:41:48.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.345+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 >> 172.21.15.94:0/1345504207 conn(0x7fd8e8066a00 msgr2=0x7fd8e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:48.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.346+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 shutdown_connections 2024-03-29T10:41:48.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.346+0000 7fd8cdffb640 1 -- 172.21.15.94:0/1345504207 wait complete. 2024-03-29T10:41:48.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:48.346+0000 7fd8cdffb640 1 librados: shutdown 2024-03-29T10:41:48.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:55.154 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:55.303 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.320+0000 7ff0cb2e1640 10 monclient: get_monmap_and_config 2024-03-29T10:41:55.303 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.320+0000 7ff0cb2e1640 10 monclient: build_initial_monmap 2024-03-29T10:41:55.303 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.320+0000 7ff0cb2e1640 10 monclient: monmap: 2024-03-29T10:41:55.303 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:55.303 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:55.303 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:41:55.320914+0000 2024-03-29T10:41:55.304 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:41:55.320914+0000 2024-03-29T10:41:55.304 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:55.304 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:55.304 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:55.304 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:41:55.304 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:41:55.304 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:55.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.320+0000 7ff0cb2e1640 1 Processor -- start 2024-03-29T10:41:55.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.320+0000 7ff0cb2e1640 1 -- start start 2024-03-29T10:41:55.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.320+0000 7ff0cb2e1640 10 monclient: init 2024-03-29T10:41:55.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.321+0000 7ff0cb2e1640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:55.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.321+0000 7ff0cb2e1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:55.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.321+0000 7ff0cb2e1640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c40b4820 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:55.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.322+0000 7ff0cb2e1640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c40a9ce0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:55.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.322+0000 7ff0cb2e1640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c40b3cd0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:55.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.322+0000 7ff0cb2e1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40b4180 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.322+0000 7ff0cb2e1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40aaf60 con 0x7ff0c40b4820 2024-03-29T10:41:55.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.322+0000 7ff0cb2e1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c413c200 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.322+0000 7ff0cb2e1640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:55.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.322+0000 7ff0cb2e1640 10 monclient(hunting): authenticate will time out at 966.590183s 2024-03-29T10:41:55.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.325+0000 7ff0caae0640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff0c40b3cd0 0x7ff0c414d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:59331/0 (socket says 172.21.15.94:59331) 2024-03-29T10:41:55.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.325+0000 7ff0caae0640 1 -- 172.21.15.94:0/1785689577 learned_addr learned my addr 172.21.15.94:0/1785689577 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:55.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189400791 0 0) 0x7ff0c413c200 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:55.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:55.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:55.310 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:41:55.310 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b0003520 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.310 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899169995 0 0) 0x7ff0b0003520 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.310 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.326+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 >> v1:172.21.15.173:6789/0 conn(0x7ff0c40b4820 legacy=0x7ff0c413baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 >> v1:172.21.15.94:6789/0 conn(0x7ff0c40a9ce0 legacy=0x7ff0c4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 1 monclient: found mon.noname-b 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4145e80 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 10 monclient: _finish_auth 0 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:55.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff0b4001690 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0c92dd640 10 monclient: dump: 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.314 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0cb2e1640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.328+0000 7ff0cb2e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0b4001860 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0c92dd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0cb2e1640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0cb2e1640 4 set_mon_vals no callback set 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0c92dd640 1 -- 172.21.15.94:0/1785689577 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff0b4001c70 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0c92dd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0c92dd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:55.315 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.329+0000 7ff0c92dd640 10 monclient: dump: 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.330+0000 7ff0cb2e1640 10 monclient: shutdown 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.330+0000 7ff0cb2e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:55.316 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.330+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/1785689577 >> v1:172.21.15.94:6790/0 conn(0x7ff0c40b3cd0 legacy=0x7ff0c414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:55.317 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.330+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/1785689577 shutdown_connections 2024-03-29T10:41:55.317 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.330+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/1785689577 >> 172.21.15.94:0/1785689577 conn(0x7ff0c406d070 msgr2=0x7ff0c4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:55.317 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.330+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/1785689577 shutdown_connections 2024-03-29T10:41:55.317 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.330+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/1785689577 wait complete. 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.331+0000 7ff0cb2e1640 10 monclient: build_initial_monmap 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.331+0000 7ff0cb2e1640 10 monclient: monmap: 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:41:55.318 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.332+0000 7ff0cb2e1640 1 librados: starting msgr at 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.332+0000 7ff0cb2e1640 1 librados: starting objecter 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.332+0000 7ff0cb2e1640 1 Processor -- start 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.332+0000 7ff0cb2e1640 1 -- start start 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.332+0000 7ff0cb2e1640 1 librados: setting wanted keys 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.332+0000 7ff0cb2e1640 1 librados: calling monclient init 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.332+0000 7ff0cb2e1640 10 monclient: init 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c40a9ce0 addr v1:172.21.15.94:6789/0 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c40b3cd0 addr v1:172.21.15.94:6790/0 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c40b4820 addr v1:172.21.15.173:6789/0 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c4129fc0 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c413b910 con 0x7ff0c40b4820 2024-03-29T10:41:55.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c413bbf0 con 0x7ff0c40b3cd0 2024-03-29T10:41:55.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:55.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.333+0000 7ff0cb2e1640 10 monclient(hunting): authenticate will time out at 966.601301s 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0caae0640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff0c40b4820 0x7ff0c41298b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:37067/0 (socket says 172.21.15.94:37067) 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0caae0640 1 -- 172.21.15.94:0/3512780025 learned_addr learned my addr 172.21.15.94:0/3512780025 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 client.?.objecter ms_handle_connect 0x7ff0c40a9ce0 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 client.?.objecter ms_handle_connect 0x7ff0c40b4820 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125025461 0 0) 0x7ff0c413b910 con 0x7ff0c40b4820 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:41:55.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.339+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a8004eb0 con 0x7ff0c40b4820 2024-03-29T10:41:55.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787072923 0 0) 0x7ff0c4129fc0 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:55.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:55.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:55.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:41:55.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c413b910 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.325 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3289568685 0 0) 0x7ff0c413b910 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c4129fc0 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff0c0001790 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): dump: 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:55.326 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3539194197 0 0) 0x7ff0c4129fc0 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 >> v1:172.21.15.94:6790/0 conn(0x7ff0c40b3cd0 legacy=0x7ff0c4135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 >> v1:172.21.15.173:6789/0 conn(0x7ff0c40b4820 legacy=0x7ff0c41298b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 monclient: found mon.a 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4144160 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:55.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0cb2e1640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0cb2e1640 10 monclient: _renew_subs 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0cb2e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.340+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c408bc10 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0cb2e1640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0cb2e1640 10 monclient: _renew_subs 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0cb2e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c4145050 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0cb2e1640 1 librados: init done 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0c0001ca0 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff0c0003790 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:55.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.341+0000 7ff0baffd640 10 monclient: dump: 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:41:55.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.344+0000 7ff0bbfff640 4 set_mon_vals no callback set 2024-03-29T10:41:55.330 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.344+0000 7ff0cb2e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:55.330 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.344+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7ff0c0019760 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.330 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.344+0000 7ff0cb2e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.344+0000 7ff0cb2e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.344+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ac005080 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.347+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7ff0c00837a0 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.347+0000 7ff0baffd640 10 client.4347.objecter ms_dispatch 0x7ff0c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.347+0000 7ff0baffd640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.347+0000 7ff0baffd640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.347+0000 7ff0baffd640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.347+0000 7ff0c9ade640 10 monclient: get_auth_request con 0x7ff0a807ed70 auth_method 0 2024-03-29T10:41:55.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.348+0000 7ff0baffd640 10 client.4347.objecter ms_handle_connect 0x7ff0a807ed70 2024-03-29T10:41:55.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.357+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0c004f080 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.357+0000 7ff0baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:55.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.357+0000 7ff0baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:55.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.684+0000 7ff0cb2e1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:55.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.684+0000 7ff0cb2e1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:55.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.684+0000 7ff0cb2e1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:41:55.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.684+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0ac005330 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.685+0000 7ff0baffd640 1 -- 172.21.15.94:0/3512780025 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7ff0c0050fb0 con 0x7ff0c40a9ce0 2024-03-29T10:41:55.669 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.685+0000 7ff0baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:55.669 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.685+0000 7ff0baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:55.669 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:41:55.669 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.688+0000 7ff0cb2e1640 10 librados: watch_flush enter 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.690+0000 7ff0cb2e1640 10 librados: watch_flush exit 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.690+0000 7ff0cb2e1640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.690+0000 7ff0cb2e1640 10 client.4347.objecter successfully canceled tick 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.690+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 >> v1:172.21.15.94:6816/3829934808 conn(0x7ff0a807ed70 legacy=0x7ff0a8081230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.690+0000 7ff0cb2e1640 10 monclient: shutdown 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.690+0000 7ff0cb2e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:55.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.690+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 >> v1:172.21.15.94:6789/0 conn(0x7ff0c40a9ce0 legacy=0x7ff0c41291a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:55.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.693+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 shutdown_connections 2024-03-29T10:41:55.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.693+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 >> 172.21.15.94:0/3512780025 conn(0x7ff0c406d070 msgr2=0x7ff0c40b72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:55.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.693+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 shutdown_connections 2024-03-29T10:41:55.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.693+0000 7ff0cb2e1640 1 -- 172.21.15.94:0/3512780025 wait complete. 2024-03-29T10:41:55.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:41:55.693+0000 7ff0cb2e1640 1 librados: shutdown 2024-03-29T10:41:55.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:02.501 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:02.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.668+0000 7f367f0fb640 10 monclient: get_monmap_and_config 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.668+0000 7f367f0fb640 10 monclient: build_initial_monmap 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.668+0000 7f367f0fb640 10 monclient: monmap: 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:02.669548+0000 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:02.669548+0000 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:02.652 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:02.653 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:02.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.670+0000 7f367f0fb640 1 Processor -- start 2024-03-29T10:42:02.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.670+0000 7f367f0fb640 1 -- start start 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.670+0000 7f367f0fb640 10 monclient: init 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f36780b1f60 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f3678071eb0 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f3678143d10 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367806ef80 con 0x7f3678071eb0 2024-03-29T10:42:02.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367806f260 con 0x7f3678143d10 2024-03-29T10:42:02.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36780b40c0 con 0x7f36780b1f60 2024-03-29T10:42:02.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:02.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.671+0000 7f367f0fb640 10 monclient(hunting): authenticate will time out at 973.939581s 2024-03-29T10:42:02.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.675+0000 7f367d698640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f3678143d10 0x7f3678146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:56787/0 (socket says 172.21.15.94:56787) 2024-03-29T10:42:02.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.675+0000 7f367d698640 1 -- 172.21.15.94:0/1889866552 learned_addr learned my addr 172.21.15.94:0/1889866552 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751246973 0 0) 0x7f367806ef80 con 0x7f3678071eb0 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3660003520 con 0x7f3678071eb0 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723275544 0 0) 0x7f367806f260 con 0x7f3678143d10 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:42:02.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.678+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f367806ef80 con 0x7f3678143d10 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2985618167 0 0) 0x7f3660003520 con 0x7f3678071eb0 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 >> v1:172.21.15.94:6790/0 conn(0x7f36780b1f60 legacy=0x7f3678143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 >> v1:172.21.15.173:6789/0 conn(0x7f3678143d10 legacy=0x7f3678146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 monclient: found mon.noname-a 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3678155d70 con 0x7f3678071eb0 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f367f0fb640 5 monclient: authenticate success, global_id 4350 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3670001a10 con 0x7f3678071eb0 2024-03-29T10:42:02.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f367f0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: dump: 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f367f0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:02.665 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3670001b70 con 0x7f3678071eb0 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f367f0fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f367f0fb640 4 set_mon_vals no callback set 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 1 -- 172.21.15.94:0/1889866552 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3670001f80 con 0x7f3678071eb0 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.679+0000 7f36777fe640 10 monclient: dump: 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:02.666 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.680+0000 7f367f0fb640 10 monclient: shutdown 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.680+0000 7f367f0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.680+0000 7f367f0fb640 1 -- 172.21.15.94:0/1889866552 >> v1:172.21.15.94:6789/0 conn(0x7f3678071eb0 legacy=0x7f3678146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.680+0000 7f367f0fb640 1 -- 172.21.15.94:0/1889866552 shutdown_connections 2024-03-29T10:42:02.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.680+0000 7f367f0fb640 1 -- 172.21.15.94:0/1889866552 >> 172.21.15.94:0/1889866552 conn(0x7f3678066a00 msgr2=0x7f3678066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:02.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.683+0000 7f367f0fb640 1 -- 172.21.15.94:0/1889866552 shutdown_connections 2024-03-29T10:42:02.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.683+0000 7f367f0fb640 1 -- 172.21.15.94:0/1889866552 wait complete. 2024-03-29T10:42:02.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 10 monclient: build_initial_monmap 2024-03-29T10:42:02.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 10 monclient: monmap: 2024-03-29T10:42:02.671 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:02.671 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:02.671 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:02.671 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:02.671 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 1 librados: starting msgr at 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 1 librados: starting objecter 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 1 Processor -- start 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 1 -- start start 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 1 librados: setting wanted keys 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 1 librados: calling monclient init 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.684+0000 7f367f0fb640 10 monclient: init 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.685+0000 7f367f0fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:02.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.685+0000 7f367f0fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:02.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.688+0000 7f367f0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f36780b1f60 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:02.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.690+0000 7f367f0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f3678143d10 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:02.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.690+0000 7f367f0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f3678070ae0 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:02.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.690+0000 7f367f0fb640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36781e7fc0 con 0x7f36780b1f60 2024-03-29T10:42:02.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.690+0000 7f367f0fb640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36781e8250 con 0x7f3678070ae0 2024-03-29T10:42:02.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.690+0000 7f367f0fb640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36781528d0 con 0x7f3678143d10 2024-03-29T10:42:02.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.690+0000 7f367f0fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:02.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.690+0000 7f367f0fb640 10 monclient(hunting): authenticate will time out at 973.958545s 2024-03-29T10:42:02.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.691+0000 7f367ce97640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f36780b1f60 0x7f36781e78b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58411/0 (socket says 172.21.15.94:58411) 2024-03-29T10:42:02.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.691+0000 7f367ce97640 1 -- 172.21.15.94:0/753985818 learned_addr learned my addr 172.21.15.94:0/753985818 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:02.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.691+0000 7f36757fa640 10 client.?.objecter ms_handle_connect 0x7f36780b1f60 2024-03-29T10:42:02.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.691+0000 7f36757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:02.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.692+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655887427 0 0) 0x7f36781e7fc0 con 0x7f36780b1f60 2024-03-29T10:42:02.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.692+0000 7f36757fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.692+0000 7f36757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.692+0000 7f36757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.692+0000 7f36757fa640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.692+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3664004eb0 con 0x7f36780b1f60 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 10 client.?.objecter ms_handle_connect 0x7f3678143d10 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 511523614 0 0) 0x7f3664004eb0 con 0x7f36780b1f60 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36781e7fc0 con 0x7f36780b1f60 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f366c001960 con 0x7f36780b1f60 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 10 monclient(hunting): dump: 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:02.678 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 455212302 0 0) 0x7f36781e7fc0 con 0x7f36780b1f60 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 >> v1:172.21.15.94:6790/0 conn(0x7f3678143d10 legacy=0x7f36780a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:02.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.693+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 >> v1:172.21.15.173:6789/0 conn(0x7f3678070ae0 legacy=0x7f36780a9e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 10 monclient: _finish_hunting 0 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 1 monclient: found mon.a 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36781f2960 con 0x7f36780b1f60 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 10 monclient: _finish_auth 0 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 10 monclient: _check_auth_tickets 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f36757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:02.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 10 monclient: _renew_subs 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36781ef930 con 0x7f36780b1f60 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 10 monclient: _renew_subs 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36781efeb0 con 0x7f36780b1f60 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.698+0000 7f367f0fb640 1 librados: init done 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f367f0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f367f0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f367f0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3678075f10 con 0x7f36780b1f60 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f366c001c40 con 0x7f36780b1f60 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f36757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:02.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f366c003a40 con 0x7f36780b1f60 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f36757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f36757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.700+0000 7f36757fa640 10 monclient: dump: 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:02.685 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:02.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.701+0000 7f3676ffd640 4 set_mon_vals no callback set 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.702+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f366c003ba0 con 0x7f36780b1f60 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.703+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f366c0849f0 con 0x7f36780b1f60 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.703+0000 7f36757fa640 10 client.4353.objecter ms_dispatch 0x7f3678000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.703+0000 7f36757fa640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.703+0000 7f36757fa640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.703+0000 7f36757fa640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.703+0000 7f3677fff640 10 monclient: get_auth_request con 0x7f366406dd50 auth_method 0 2024-03-29T10:42:02.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.703+0000 7f36757fa640 10 client.4353.objecter ms_handle_connect 0x7f366406dd50 2024-03-29T10:42:02.692 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.708+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f366c0502d0 con 0x7f36780b1f60 2024-03-29T10:42:02.692 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.709+0000 7f36757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:02.692 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:02.709+0000 7f36757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:03.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.024+0000 7f367f0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:03.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.024+0000 7f367f0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:03.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.024+0000 7f367f0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:42:03.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.024+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36780b69d0 con 0x7f36780b1f60 2024-03-29T10:42:03.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.025+0000 7f36757fa640 1 -- 172.21.15.94:0/753985818 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f366c052200 con 0x7f36780b1f60 2024-03-29T10:42:03.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.025+0000 7f36757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:03.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.025+0000 7f36757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:03.010 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:03.010 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:03.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.030+0000 7f367f0fb640 10 librados: watch_flush enter 2024-03-29T10:42:03.015 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 10 librados: watch_flush exit 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 10 client.4353.objecter successfully canceled tick 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 >> v1:172.21.15.94:6816/3829934808 conn(0x7f366406dd50 legacy=0x7f3664070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 10 monclient: shutdown 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 >> v1:172.21.15.94:6789/0 conn(0x7f36780b1f60 legacy=0x7f36781e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 shutdown_connections 2024-03-29T10:42:03.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.032+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 >> 172.21.15.94:0/753985818 conn(0x7f3678066a00 msgr2=0x7f3678142900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:03.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.033+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 shutdown_connections 2024-03-29T10:42:03.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.033+0000 7f367f0fb640 1 -- 172.21.15.94:0/753985818 wait complete. 2024-03-29T10:42:03.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:03.033+0000 7f367f0fb640 1 librados: shutdown 2024-03-29T10:42:03.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:09.843 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:09.999 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.016+0000 7f178716b640 10 monclient: get_monmap_and_config 2024-03-29T10:42:09.999 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.016+0000 7f178716b640 10 monclient: build_initial_monmap 2024-03-29T10:42:09.999 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.016+0000 7f178716b640 10 monclient: monmap: 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:10.017012+0000 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:10.017012+0000 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:10.000 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:10.001 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.017+0000 7f178716b640 1 Processor -- start 2024-03-29T10:42:10.001 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.017+0000 7f178716b640 1 -- start start 2024-03-29T10:42:10.001 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.017+0000 7f178716b640 10 monclient: init 2024-03-29T10:42:10.001 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:10.001 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:10.001 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 10 monclient(hunting): picked mon.noname-b con 0x7f17800b2800 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:10.002 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 10 monclient(hunting): picked mon.noname-c con 0x7f17800a9c00 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:10.002 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 10 monclient(hunting): picked mon.noname-a con 0x7f17800b38c0 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:10.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1780141eb0 con 0x7f17800b38c0 2024-03-29T10:42:10.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17800b22d0 con 0x7f17800a9c00 2024-03-29T10:42:10.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.018+0000 7f178716b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1780142d30 con 0x7f17800b2800 2024-03-29T10:42:10.007 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785968640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f17800a9c00 0x7f178013bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45252/0 (socket says 172.21.15.94:45252) 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785968640 1 -- 172.21.15.94:0/560804516 learned_addr learned my addr 172.21.15.94:0/560804516 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f178716b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f178716b640 10 monclient(hunting): authenticate will time out at 981.287263s 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378242932 0 0) 0x7f17800b22d0 con 0x7f17800a9c00 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785167640 10 monclient(hunting): _negotiate 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785167640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785167640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785167640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.019+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c003520 con 0x7f17800a9c00 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436567955 0 0) 0x7f1780141eb0 con 0x7f17800b38c0 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): _negotiate 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:10.008 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17800b22d0 con 0x7f17800b38c0 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787768471 0 0) 0x7f1780142d30 con 0x7f17800b2800 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): _negotiate 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1780141eb0 con 0x7f17800b2800 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.020+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556808738 0 0) 0x7f17800b22d0 con 0x7f17800b38c0 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 >> v1:172.21.15.94:6790/0 conn(0x7f17800b2800 legacy=0x7f1780142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 >> v1:172.21.15.173:6789/0 conn(0x7f17800a9c00 legacy=0x7f178013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 10 monclient: _finish_hunting 0 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 1 monclient: found mon.noname-a 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:10.009 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17800aaf90 con 0x7f17800b38c0 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 10 monclient: _finish_auth 0 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 10 monclient: _check_auth_tickets 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f178716b640 5 monclient: authenticate success, global_id 4356 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f178716b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f17780019b0 con 0x7f17800b38c0 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:10.010 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f1785167640 10 monclient: dump: 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.021+0000 7f178716b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1778001b90 con 0x7f17800b38c0 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f1785167640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f1785167640 1 -- 172.21.15.94:0/560804516 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1778001f20 con 0x7f17800b38c0 2024-03-29T10:42:10.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f1785167640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f1785167640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f1785167640 10 monclient: dump: 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f178716b640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:10.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.023+0000 7f178716b640 4 set_mon_vals no callback set 2024-03-29T10:42:10.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.024+0000 7f178716b640 10 monclient: shutdown 2024-03-29T10:42:10.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.024+0000 7f178716b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:10.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.024+0000 7f178716b640 1 -- 172.21.15.94:0/560804516 >> v1:172.21.15.94:6789/0 conn(0x7f17800b38c0 legacy=0x7f178014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:10.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.024+0000 7f178716b640 1 -- 172.21.15.94:0/560804516 shutdown_connections 2024-03-29T10:42:10.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.024+0000 7f178716b640 1 -- 172.21.15.94:0/560804516 >> 172.21.15.94:0/560804516 conn(0x7f1780066a00 msgr2=0x7f1780066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:10.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.024+0000 7f178716b640 1 -- 172.21.15.94:0/560804516 shutdown_connections 2024-03-29T10:42:10.013 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.024+0000 7f178716b640 1 -- 172.21.15.94:0/560804516 wait complete. 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.025+0000 7f178716b640 10 monclient: build_initial_monmap 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.025+0000 7f178716b640 10 monclient: monmap: 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:10.016 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.026+0000 7f178716b640 1 librados: starting msgr at 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.026+0000 7f178716b640 1 librados: starting objecter 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.026+0000 7f178716b640 1 Processor -- start 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.026+0000 7f178716b640 1 -- start start 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.026+0000 7f178716b640 1 librados: setting wanted keys 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.026+0000 7f178716b640 1 librados: calling monclient init 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.026+0000 7f178716b640 10 monclient: init 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 10 monclient(hunting): picked mon.noname-c con 0x7f17800a9c00 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 10 monclient(hunting): picked mon.noname-b con 0x7f17800b2800 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f1786169640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f17800a9c00 0x7f17801d3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:60484/0 (socket says 172.21.15.94:60484) 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f1786169640 1 -- 172.21.15.94:0/1002827057 learned_addr learned my addr 172.21.15.94:0/1002827057 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:10.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 10 monclient(hunting): picked mon.noname-a con 0x7f17800b38c0 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17801d2d00 con 0x7f17800b38c0 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f1776ffd640 10 client.?.objecter ms_handle_connect 0x7f17800a9c00 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f1776ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17801d38a0 con 0x7f17800b2800 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17801d2990 con 0x7f17800a9c00 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f1776ffd640 10 client.?.objecter ms_handle_connect 0x7f17800b2800 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f1776ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.027+0000 7f178716b640 10 monclient(hunting): authenticate will time out at 981.295921s 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 client.?.objecter ms_handle_connect 0x7f17800b38c0 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001569966 0 0) 0x7f17801d38a0 con 0x7f17800b2800 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:10.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1764004eb0 con 0x7f17800b2800 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634775113 0 0) 0x7f17801d2990 con 0x7f17800a9c00 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17801d38a0 con 0x7f17800a9c00 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1342810208 0 0) 0x7f1764004eb0 con 0x7f17800b2800 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17801d2990 con 0x7f17800b2800 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1770001800 con 0x7f17800b2800 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:10.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.028+0000 7f1776ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1238442459 0 0) 0x7f17801d2990 con 0x7f17800b2800 2024-03-29T10:42:10.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 >> v1:172.21.15.94:6790/0 conn(0x7f17800a9c00 legacy=0x7f17801d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 >> v1:172.21.15.94:6789/0 conn(0x7f17800b38c0 legacy=0x7f17801dd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 1 monclient: found mon.b 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1780070170 con 0x7f17800b2800 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 10 monclient: _renew_subs 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1780071420 con 0x7f17800b2800 2024-03-29T10:42:10.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 10 monclient: _renew_subs 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f178006e070 con 0x7f17800b2800 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f178716b640 1 librados: init done 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1770001c10 con 0x7f17800b2800 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1770004980 con 0x7f17800b2800 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1776ffd640 10 monclient: dump: 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.022 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.029+0000 7f1777fff640 4 set_mon_vals no callback set 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.030+0000 7f178716b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.030+0000 7f178716b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.030+0000 7f178716b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.030+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1768005080 con 0x7f17800b2800 2024-03-29T10:42:10.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.031+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f1770004ae0 con 0x7f17800b2800 2024-03-29T10:42:10.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.032+0000 7f1786169640 10 monclient: get_auth_request con 0x7f176406dd10 auth_method 0 2024-03-29T10:42:10.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.033+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f1770084960 con 0x7f17800b2800 2024-03-29T10:42:10.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.033+0000 7f1776ffd640 10 client.4336.objecter ms_dispatch 0x7f1780000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:10.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.033+0000 7f1776ffd640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:10.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.033+0000 7f1776ffd640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:10.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.033+0000 7f1776ffd640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T10:42:10.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.033+0000 7f1776ffd640 10 client.4336.objecter ms_handle_connect 0x7f176406dd10 2024-03-29T10:42:10.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.038+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1770050240 con 0x7f17800b2800 2024-03-29T10:42:10.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.038+0000 7f1776ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:10.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.038+0000 7f1776ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:10.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.495+0000 7f178716b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:10.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.495+0000 7f178716b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:10.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.495+0000 7f178716b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:42:10.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.495+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1768005600 con 0x7f17800b2800 2024-03-29T10:42:10.479 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.495+0000 7f1776ffd640 1 -- 172.21.15.94:0/1002827057 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f1770052170 con 0x7f17800b2800 2024-03-29T10:42:10.479 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.495+0000 7f1776ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:10.479 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.495+0000 7f1776ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:10.479 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:10.480 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:10.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 10 librados: watch_flush enter 2024-03-29T10:42:10.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 10 librados: watch_flush exit 2024-03-29T10:42:10.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T10:42:10.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 10 client.4336.objecter successfully canceled tick 2024-03-29T10:42:10.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 >> v1:172.21.15.94:6816/3829934808 conn(0x7f176406dd10 legacy=0x7f17640701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:10.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 10 monclient: shutdown 2024-03-29T10:42:10.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:10.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 >> v1:172.21.15.173:6789/0 conn(0x7f17800b2800 legacy=0x7f1780070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:10.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 shutdown_connections 2024-03-29T10:42:10.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.499+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 >> 172.21.15.94:0/1002827057 conn(0x7f1780066a00 msgr2=0x7f178013bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:10.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.500+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 shutdown_connections 2024-03-29T10:42:10.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.500+0000 7f178716b640 1 -- 172.21.15.94:0/1002827057 wait complete. 2024-03-29T10:42:10.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:10.500+0000 7f178716b640 1 librados: shutdown 2024-03-29T10:42:10.499 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:17.306 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:17.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.472+0000 7ff2a4bc6640 10 monclient: get_monmap_and_config 2024-03-29T10:42:17.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.472+0000 7ff2a4bc6640 10 monclient: build_initial_monmap 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.472+0000 7ff2a4bc6640 10 monclient: monmap: 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:17.473317+0000 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:17.473317+0000 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:17.456 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:17.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.473+0000 7ff2a4bc6640 1 Processor -- start 2024-03-29T10:42:17.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.473+0000 7ff2a4bc6640 1 -- start start 2024-03-29T10:42:17.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.473+0000 7ff2a4bc6640 10 monclient: init 2024-03-29T10:42:17.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.474+0000 7ff2a4bc6640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:17.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.474+0000 7ff2a4bc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:17.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.474+0000 7ff2a4bc6640 10 monclient(hunting): picked mon.noname-b con 0x7ff2a00b0770 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:17.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.474+0000 7ff2a4bc6640 10 monclient(hunting): picked mon.noname-a con 0x7ff2a014a700 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:17.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff2a4bc6640 10 monclient(hunting): picked mon.noname-c con 0x7ff2a00ac5a0 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:17.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff2a4bc6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a00aba70 con 0x7ff2a014a700 2024-03-29T10:42:17.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff2a4bc6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a00aecb0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29ed9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff2a00ac5a0 0x7ff2a0151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:55751/0 (socket says 172.21.15.94:55751) 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29ed9d640 1 -- 172.21.15.94:0/3695428947 learned_addr learned my addr 172.21.15.94:0/3695428947 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3695428947 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a00af150 con 0x7ff2a00b0770 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff2a4bc6640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff2a4bc6640 10 monclient(hunting): authenticate will time out at 988.745511s 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403340694 0 0) 0x7ff2a00aecb0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29d59a640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:42:17.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.477+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff288003520 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.478+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2689333582 0 0) 0x7ff288003520 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.478+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 >> v1:172.21.15.94:6790/0 conn(0x7ff2a00b0770 legacy=0x7ff2a0149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:17.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.478+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 >> v1:172.21.15.94:6789/0 conn(0x7ff2a014a700 legacy=0x7ff2a00abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 1 monclient: found mon.noname-c 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00ac9d0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 10 monclient: _finish_auth 0 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff290001a30 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.481+0000 7ff29d59a640 10 monclient: dump: 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:17.465 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff2a4bc6640 5 monclient: authenticate success, global_id 4339 2024-03-29T10:42:17.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff2a4bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff290001c00 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff29d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff29d59a640 1 -- 172.21.15.94:0/3695428947 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff290002050 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff29d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff29d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff29d59a640 10 monclient: dump: 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:17.467 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff2a4bc6640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.482+0000 7ff2a4bc6640 4 set_mon_vals no callback set 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.483+0000 7ff2a4bc6640 10 monclient: shutdown 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.483+0000 7ff2a4bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.483+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3695428947 >> v1:172.21.15.173:6789/0 conn(0x7ff2a00ac5a0 legacy=0x7ff2a0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.483+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3695428947 shutdown_connections 2024-03-29T10:42:17.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.483+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3695428947 >> 172.21.15.94:0/3695428947 conn(0x7ff2a006b660 msgr2=0x7ff2a006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:17.469 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.483+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3695428947 shutdown_connections 2024-03-29T10:42:17.469 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.483+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3695428947 wait complete. 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.485+0000 7ff2a4bc6640 10 monclient: build_initial_monmap 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.485+0000 7ff2a4bc6640 10 monclient: monmap: 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:17.470 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.485+0000 7ff2a4bc6640 1 librados: starting msgr at 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.485+0000 7ff2a4bc6640 1 librados: starting objecter 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.486+0000 7ff2a4bc6640 1 Processor -- start 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.486+0000 7ff2a4bc6640 1 -- start start 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.486+0000 7ff2a4bc6640 1 librados: setting wanted keys 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.486+0000 7ff2a4bc6640 1 librados: calling monclient init 2024-03-29T10:42:17.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.486+0000 7ff2a4bc6640 10 monclient: init 2024-03-29T10:42:17.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.490+0000 7ff2a4bc6640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:17.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.490+0000 7ff2a4bc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:17.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.490+0000 7ff2a4bc6640 10 monclient(hunting): picked mon.noname-c con 0x7ff2a00ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:17.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff2a4bc6640 10 monclient(hunting): picked mon.noname-a con 0x7ff2a00b0770 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:17.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff2a4bc6640 10 monclient(hunting): picked mon.noname-b con 0x7ff2a014a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:17.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff2a4bc6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a01284a0 con 0x7ff2a00b0770 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff2a4bc6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a01db8c0 con 0x7ff2a014a700 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff2a4bc6640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a0126090 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff2a4bc6640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff2a4bc6640 10 monclient(hunting): authenticate will time out at 988.759900s 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff29e59c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff2a00ac5a0 0x7ff2a01dbde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:45478/0 (socket says 172.21.15.94:45478) 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.491+0000 7ff29e59c640 1 -- 172.21.15.94:0/3989058496 learned_addr learned my addr 172.21.15.94:0/3989058496 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.492+0000 7ff28effd640 10 client.?.objecter ms_handle_connect 0x7ff2a00ac5a0 2024-03-29T10:42:17.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.492+0000 7ff28effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493853685 0 0) 0x7ff2a0126090 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280004eb0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976602931 0 0) 0x7ff280004eb0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2a0126090 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff298001960 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:17.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): dump: 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.493+0000 7ff28effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.494+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4135493279 0 0) 0x7ff2a0126090 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.494+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 >> v1:172.21.15.173:6789/0 conn(0x7ff2a014a700 legacy=0x7ff2a01e7090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:17.479 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.496+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 >> v1:172.21.15.94:6789/0 conn(0x7ff2a00b0770 legacy=0x7ff2a0145190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:17.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:17.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 1 monclient: found mon.c 2024-03-29T10:42:17.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:17.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:17.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a01239d0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 10 monclient: _finish_auth 0 2024-03-29T10:42:17.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:17.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.499+0000 7ff28effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:17.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 5 monclient: authenticate success, global_id 4349 2024-03-29T10:42:17.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 10 monclient: _renew_subs 2024-03-29T10:42:17.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:17.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a01dc4f0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:17.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 10 monclient: _renew_subs 2024-03-29T10:42:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a0127660 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.501+0000 7ff2a4bc6640 1 librados: init done 2024-03-29T10:42:17.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.502+0000 7ff2a4bc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:17.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.502+0000 7ff2a4bc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:17.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.502+0000 7ff2a4bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:17.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.502+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a0123bc0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.505+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff298001c80 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.505+0000 7ff28effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.505+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff298003a40 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.505+0000 7ff28effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.505+0000 7ff28effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.505+0000 7ff28effd640 10 monclient: dump: 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:17.489 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.505+0000 7ff29cd99640 4 set_mon_vals no callback set 2024-03-29T10:42:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.507+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7ff298003ba0 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.507+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7ff298084a90 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.507+0000 7ff28effd640 10 client.4349.objecter ms_dispatch 0x7ff2a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.507+0000 7ff28effd640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.507+0000 7ff28effd640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.507+0000 7ff29dd9b640 10 monclient: get_auth_request con 0x7ff28006db30 auth_method 0 2024-03-29T10:42:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.507+0000 7ff28effd640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T10:42:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.508+0000 7ff28effd640 10 client.4349.objecter ms_handle_connect 0x7ff28006db30 2024-03-29T10:42:17.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.513+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff298050370 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.513+0000 7ff28effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:17.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.513+0000 7ff28effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:17.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.833+0000 7ff2a4bc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:17.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.834+0000 7ff2a4bc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:17.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.834+0000 7ff2a4bc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:17.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.834+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2a01dc680 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.835+0000 7ff28effd640 1 -- 172.21.15.94:0/3989058496 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7ff298050370 con 0x7ff2a00ac5a0 2024-03-29T10:42:17.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.835+0000 7ff28effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:17.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.835+0000 7ff28effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:17.821 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:17.821 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 10 librados: watch_flush enter 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 10 librados: watch_flush exit 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 10 client.4349.objecter successfully canceled tick 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 >> v1:172.21.15.94:6816/3829934808 conn(0x7ff28006db30 legacy=0x7ff28006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 10 monclient: shutdown 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 >> v1:172.21.15.94:6790/0 conn(0x7ff2a00ac5a0 legacy=0x7ff2a01dbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 shutdown_connections 2024-03-29T10:42:17.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.839+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 >> 172.21.15.94:0/3989058496 conn(0x7ff2a006b660 msgr2=0x7ff2a006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:17.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.841+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 shutdown_connections 2024-03-29T10:42:17.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.843+0000 7ff2a4bc6640 1 -- 172.21.15.94:0/3989058496 wait complete. 2024-03-29T10:42:17.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:17.843+0000 7ff2a4bc6640 1 librados: shutdown 2024-03-29T10:42:17.843 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:24.650 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:24.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.817+0000 7f1dcce9c640 10 monclient: get_monmap_and_config 2024-03-29T10:42:24.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.817+0000 7f1dcce9c640 10 monclient: build_initial_monmap 2024-03-29T10:42:24.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.817+0000 7f1dcce9c640 10 monclient: monmap: 2024-03-29T10:42:24.800 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:24.818246+0000 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:24.818246+0000 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:24.801 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.802 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.818+0000 7f1dcce9c640 1 Processor -- start 2024-03-29T10:42:24.802 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.818+0000 7f1dcce9c640 1 -- start start 2024-03-29T10:42:24.802 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.818+0000 7f1dcce9c640 10 monclient: init 2024-03-29T10:42:24.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.819+0000 7f1dcce9c640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.819+0000 7f1dcce9c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.819+0000 7f1dcce9c640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc80b4710 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.819+0000 7f1dcce9c640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc80b3970 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.819+0000 7f1dcce9c640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc813f390 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dcce9c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc80af020 con 0x7f1dc80b3970 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dcce9c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc80b0910 con 0x7f1dc80b4710 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dcce9c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc80b0bf0 con 0x7f1dc813f390 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dcce9c640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc5d9b640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f1dc80b3970 0x7f1dc80af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42180/0 (socket says 172.21.15.94:42180) 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc5d9b640 1 -- 172.21.15.94:0/448600813 learned_addr learned my addr 172.21.15.94:0/448600813 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dcce9c640 10 monclient(hunting): authenticate will time out at 996.088543s 2024-03-29T10:42:24.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247862449 0 0) 0x7f1dc80b0910 con 0x7f1dc80b4710 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc559a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc559a640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.820+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0003520 con 0x7f1dc80b4710 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175904808 0 0) 0x7f1dc80af020 con 0x7f1dc80b3970 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc80b0910 con 0x7f1dc80b3970 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176977556 0 0) 0x7f1dc80b0bf0 con 0x7f1dc813f390 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:24.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc80af020 con 0x7f1dc813f390 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 26327023 0 0) 0x7f1db0003520 con 0x7f1dc80b4710 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 >> v1:172.21.15.94:6790/0 conn(0x7f1dc813f390 legacy=0x7f1dc813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 >> v1:172.21.15.94:6789/0 conn(0x7f1dc80b3970 legacy=0x7f1dc80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 monclient: found mon.noname-c 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc80aeb00 con 0x7f1dc80b4710 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient: _finish_auth 0 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:24.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1db8001610 con 0x7f1dc80b4710 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dc559a640 10 monclient: dump: 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dcce9c640 5 monclient: authenticate success, global_id 4342 2024-03-29T10:42:24.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.821+0000 7f1dcce9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1db8001770 con 0x7f1dc80b4710 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dc559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dc559a640 1 -- 172.21.15.94:0/448600813 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1db8001bc0 con 0x7f1dc80b4710 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dc559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dc559a640 10 monclient: dump: 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:24.814 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dcce9c640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.822+0000 7f1dcce9c640 4 set_mon_vals no callback set 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.823+0000 7f1dcce9c640 10 monclient: shutdown 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.823+0000 7f1dcce9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.823+0000 7f1dcce9c640 1 -- 172.21.15.94:0/448600813 >> v1:172.21.15.173:6789/0 conn(0x7f1dc80b4710 legacy=0x7f1dc80b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.823+0000 7f1dcce9c640 1 -- 172.21.15.94:0/448600813 shutdown_connections 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.823+0000 7f1dcce9c640 1 -- 172.21.15.94:0/448600813 >> 172.21.15.94:0/448600813 conn(0x7f1dc8066a00 msgr2=0x7f1dc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.823+0000 7f1dcce9c640 1 -- 172.21.15.94:0/448600813 shutdown_connections 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.824+0000 7f1dcce9c640 1 -- 172.21.15.94:0/448600813 wait complete. 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.825+0000 7f1dcce9c640 10 monclient: build_initial_monmap 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.825+0000 7f1dcce9c640 10 monclient: monmap: 2024-03-29T10:42:24.815 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:24.816 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:24.816 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:24.816 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:24.816 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:24.816 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.816 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:24.816 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.826+0000 7f1dcce9c640 1 librados: starting msgr at 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.826+0000 7f1dcce9c640 1 librados: starting objecter 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.826+0000 7f1dcce9c640 1 Processor -- start 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.826+0000 7f1dcce9c640 1 -- start start 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.826+0000 7f1dcce9c640 1 librados: setting wanted keys 2024-03-29T10:42:24.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.826+0000 7f1dcce9c640 1 librados: calling monclient init 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.826+0000 7f1dcce9c640 10 monclient: init 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 10 monclient(hunting): picked mon.noname-a con 0x7f1dc80b3970 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 10 monclient(hunting): picked mon.noname-b con 0x7f1dc80b4710 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 10 monclient(hunting): picked mon.noname-c con 0x7f1dc813f390 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dc5d9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f1dc80b4710 0x7f1dc8132830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45728/0 (socket says 172.21.15.94:45728) 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dc5d9b640 1 -- 172.21.15.94:0/665925865 learned_addr learned my addr 172.21.15.94:0/665925865 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc8133650 con 0x7f1dc80b3970 2024-03-29T10:42:24.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc806ee30 con 0x7f1dc80b4710 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc806f110 con 0x7f1dc813f390 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1db6ffd640 10 client.?.objecter ms_handle_connect 0x7f1dc80b4710 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.827+0000 7f1dcce9c640 10 monclient(hunting): authenticate will time out at 996.096077s 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 client.?.objecter ms_handle_connect 0x7f1dc80b3970 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 client.?.objecter ms_handle_connect 0x7f1dc813f390 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:24.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178907782 0 0) 0x7f1dc806ee30 con 0x7f1dc80b4710 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da8004eb0 con 0x7f1dc80b4710 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073720617 0 0) 0x7f1dc8133650 con 0x7f1dc80b3970 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.828+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc806ee30 con 0x7f1dc80b3970 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345820948 0 0) 0x7f1dc806f110 con 0x7f1dc813f390 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:24.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc8133650 con 0x7f1dc813f390 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358073622 0 0) 0x7f1dc8133650 con 0x7f1dc813f390 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc806f110 con 0x7f1dc813f390 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1dc00019c0 con 0x7f1dc813f390 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:24.821 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425614777 0 0) 0x7f1dc806ee30 con 0x7f1dc80b3970 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc8133650 con 0x7f1dc80b3970 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1db8001a80 con 0x7f1dc80b3970 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.829+0000 7f1db6ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:24.822 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 618143086 0 0) 0x7f1da8004eb0 con 0x7f1dc80b4710 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dc806ee30 con 0x7f1dc80b4710 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1dbc0019f0 con 0x7f1dc80b4710 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:24.823 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2851988460 0 0) 0x7f1dc806f110 con 0x7f1dc813f390 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 >> v1:172.21.15.173:6789/0 conn(0x7f1dc80b4710 legacy=0x7f1dc8132830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 >> v1:172.21.15.94:6789/0 conn(0x7f1dc80b3970 legacy=0x7f1dc806fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:24.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:24.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 monclient: found mon.c 2024-03-29T10:42:24.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:24.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:24.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc806f3f0 con 0x7f1dc813f390 2024-03-29T10:42:24.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:24.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 10 monclient: _renew_subs 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc80ae1b0 con 0x7f1dc813f390 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 10 monclient: _renew_subs 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc80ac4e0 con 0x7f1dc813f390 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.830+0000 7f1dcce9c640 1 librados: init done 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.831+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dc0001ed0 con 0x7f1dc813f390 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.831+0000 7f1db6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.831+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1dc0003b30 con 0x7f1dc813f390 2024-03-29T10:42:24.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.831+0000 7f1db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.831+0000 7f1db6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.831+0000 7f1db6ffd640 10 monclient: dump: 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.831+0000 7f1dc4d99640 4 set_mon_vals no callback set 2024-03-29T10:42:24.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.832+0000 7f1dcce9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.832+0000 7f1dcce9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.832+0000 7f1dcce9c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.832+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac005080 con 0x7f1dc813f390 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.834+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f1dc0019b00 con 0x7f1dc813f390 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.834+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f1dc0083980 con 0x7f1dc813f390 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.834+0000 7f1db6ffd640 10 client.4355.objecter ms_dispatch 0x7f1dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.834+0000 7f1db6ffd640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.834+0000 7f1db6ffd640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.834+0000 7f1db6ffd640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.835+0000 7f1dc659c640 10 monclient: get_auth_request con 0x7f1da806e0f0 auth_method 0 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.835+0000 7f1db6ffd640 10 client.4355.objecter ms_handle_connect 0x7f1da806e0f0 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.842+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dc004f260 con 0x7f1dc813f390 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.842+0000 7f1db6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:24.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:24.842+0000 7f1db6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:25.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.226+0000 7f1dcce9c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:25.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.226+0000 7f1dcce9c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:25.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.226+0000 7f1dcce9c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:25.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.226+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dac005330 con 0x7f1dc813f390 2024-03-29T10:42:25.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.228+0000 7f1db6ffd640 1 -- 172.21.15.94:0/665925865 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f1dc0051190 con 0x7f1dc813f390 2024-03-29T10:42:25.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.228+0000 7f1db6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:25.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.228+0000 7f1db6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:25.212 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:25.212 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:25.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.231+0000 7f1dcce9c640 10 librados: watch_flush enter 2024-03-29T10:42:25.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.231+0000 7f1dcce9c640 10 librados: watch_flush exit 2024-03-29T10:42:25.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.231+0000 7f1dcce9c640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.231+0000 7f1dcce9c640 10 client.4355.objecter successfully canceled tick 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 >> v1:172.21.15.94:6816/3829934808 conn(0x7f1da806e0f0 legacy=0x7f1da80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 10 monclient: shutdown 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 >> v1:172.21.15.94:6790/0 conn(0x7f1dc813f390 legacy=0x7f1dc8132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 shutdown_connections 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 >> 172.21.15.94:0/665925865 conn(0x7f1dc8066a00 msgr2=0x7f1dc80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 shutdown_connections 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 1 -- 172.21.15.94:0/665925865 wait complete. 2024-03-29T10:42:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:25.232+0000 7f1dcce9c640 1 librados: shutdown 2024-03-29T10:42:25.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:32.039 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:32.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.191+0000 7f692bc63640 10 monclient: get_monmap_and_config 2024-03-29T10:42:32.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.191+0000 7f692bc63640 10 monclient: build_initial_monmap 2024-03-29T10:42:32.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.191+0000 7f692bc63640 10 monclient: monmap: 2024-03-29T10:42:32.174 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:32.191957+0000 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:32.191957+0000 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:32.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.192+0000 7f692bc63640 1 Processor -- start 2024-03-29T10:42:32.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.192+0000 7f692bc63640 1 -- start start 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.192+0000 7f692bc63640 10 monclient: init 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 10 monclient(hunting): picked mon.noname-c con 0x7f69241457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 10 monclient(hunting): picked mon.noname-b con 0x7f692413f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 10 monclient(hunting): picked mon.noname-a con 0x7f69240a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69240aaef0 con 0x7f69240a9d70 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6924145bd0 con 0x7f69241457f0 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6924145eb0 con 0x7f692413f650 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.193+0000 7f692bc63640 10 monclient(hunting): authenticate will time out at 1003.461584s 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69299ff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f69241457f0 0x7f692413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:51330/0 (socket says 172.21.15.94:51330) 2024-03-29T10:42:32.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69299ff640 1 -- 172.21.15.94:0/58694654 learned_addr learned my addr 172.21.15.94:0/58694654 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489740172 0 0) 0x7f6924145bd0 con 0x7f69241457f0 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6910003520 con 0x7f69241457f0 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292111051 0 0) 0x7f6910003520 con 0x7f69241457f0 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 >> v1:172.21.15.94:6790/0 conn(0x7f692413f650 legacy=0x7f69240b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:32.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.194+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 >> v1:172.21.15.94:6789/0 conn(0x7f69240a9d70 legacy=0x7f692414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:32.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 1 monclient: found mon.noname-c 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924146340 con 0x7f69241457f0 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: _finish_auth 0 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6920001690 con 0x7f69241457f0 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f692bc63640 5 monclient: authenticate success, global_id 4348 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f692bc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:32.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: dump: 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f692bc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69200018f0 con 0x7f69241457f0 2024-03-29T10:42:32.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 1 -- 172.21.15.94:0/58694654 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6920001d40 con 0x7f69241457f0 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f69289fd640 10 monclient: dump: 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:32.185 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f692bc63640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.196+0000 7f692bc63640 4 set_mon_vals no callback set 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.197+0000 7f692bc63640 10 monclient: shutdown 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.197+0000 7f692bc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.197+0000 7f692bc63640 1 -- 172.21.15.94:0/58694654 >> v1:172.21.15.173:6789/0 conn(0x7f69241457f0 legacy=0x7f692413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.197+0000 7f692bc63640 1 -- 172.21.15.94:0/58694654 shutdown_connections 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.197+0000 7f692bc63640 1 -- 172.21.15.94:0/58694654 >> 172.21.15.94:0/58694654 conn(0x7f6924066a00 msgr2=0x7f6924066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.197+0000 7f692bc63640 1 -- 172.21.15.94:0/58694654 shutdown_connections 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.197+0000 7f692bc63640 1 -- 172.21.15.94:0/58694654 wait complete. 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.198+0000 7f692bc63640 10 monclient: build_initial_monmap 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.198+0000 7f692bc63640 10 monclient: monmap: 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:32.186 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 1 librados: starting msgr at 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 1 librados: starting objecter 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 1 Processor -- start 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 1 -- start start 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 1 librados: setting wanted keys 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 1 librados: calling monclient init 2024-03-29T10:42:32.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 10 monclient: init 2024-03-29T10:42:32.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:32.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:32.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f692bc63640 10 monclient(hunting): picked mon.noname-c con 0x7f69240a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.199+0000 7f69299ff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f69240a9d70 0x7f6924142160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36202/0 (socket says 172.21.15.94:36202) 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f69299ff640 1 -- 172.21.15.94:0/410578855 learned_addr learned my addr 172.21.15.94:0/410578855 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f692bc63640 10 monclient(hunting): picked mon.noname-a con 0x7f692413f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f692bc63640 10 monclient(hunting): picked mon.noname-b con 0x7f69241457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f692bc63640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69241d6ff0 con 0x7f692413f650 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f692bc63640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6924142870 con 0x7f69241457f0 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f692bc63640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692413fda0 con 0x7f69240a9d70 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f692bc63640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f692bc63640 10 monclient(hunting): authenticate will time out at 1003.468410s 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f691a7fc640 10 client.?.objecter ms_handle_connect 0x7f69240a9d70 2024-03-29T10:42:32.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f691a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f691a7fc640 10 client.?.objecter ms_handle_connect 0x7f69241457f0 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f691a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.200+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294840499 0 0) 0x7f692413fda0 con 0x7f69240a9d70 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6908004eb0 con 0x7f69240a9d70 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488663798 0 0) 0x7f6924142870 con 0x7f69241457f0 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:42:32.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692413fda0 con 0x7f69241457f0 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218179774 0 0) 0x7f692413fda0 con 0x7f69241457f0 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6908005490 con 0x7f69241457f0 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3799655954 0 0) 0x7f6908004eb0 con 0x7f69240a9d70 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6908005a70 con 0x7f69240a9d70 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6920001650 con 0x7f69240a9d70 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): dump: 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.194 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f691c001b10 con 0x7f69241457f0 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.201+0000 7f691a7fc640 10 monclient(hunting): dump: 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.195 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.202+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1373164848 0 0) 0x7f6908005a70 con 0x7f69240a9d70 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.202+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 >> v1:172.21.15.173:6789/0 conn(0x7f69241457f0 legacy=0x7f69241d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.202+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 >> v1:172.21.15.94:6789/0 conn(0x7f692413f650 legacy=0x7f692412dca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 1 monclient: found mon.c 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:32.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69241417e0 con 0x7f69240a9d70 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f691a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 5 monclient: authenticate success, global_id 4358 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 10 monclient: _renew_subs 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692412e3b0 con 0x7f69240a9d70 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 10 monclient: _renew_subs 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69241d7d70 con 0x7f69240a9d70 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.203+0000 7f692bc63640 1 librados: init done 2024-03-29T10:42:32.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.205+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6920004a90 con 0x7f69240a9d70 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.205+0000 7f691a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.205+0000 7f690ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.205+0000 7f691bfff640 4 set_mon_vals no callback set 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6920004ea0 con 0x7f69240a9d70 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f690ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f690ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692412e750 con 0x7f69240a9d70 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f691a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f691a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f691a7fc640 10 monclient: dump: 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:32.198 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.206+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f6920005080 con 0x7f69240a9d70 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.207+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f6920084ca0 con 0x7f69240a9d70 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.207+0000 7f691a7fc640 10 client.4358.objecter ms_dispatch 0x7f6924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.207+0000 7f691a7fc640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:32.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.207+0000 7f691a7fc640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:32.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.207+0000 7f691a7fc640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T10:42:32.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.207+0000 7f69291fe640 10 monclient: get_auth_request con 0x7f690806e390 auth_method 0 2024-03-29T10:42:32.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.208+0000 7f691a7fc640 10 client.4358.objecter ms_handle_connect 0x7f690806e390 2024-03-29T10:42:32.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.212+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f692412e750 con 0x7f69240a9d70 2024-03-29T10:42:32.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.212+0000 7f691a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:32.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.212+0000 7f691a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:32.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.526+0000 7f690ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:32.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.526+0000 7f690ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:32.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.526+0000 7f690ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:32.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.526+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f692412e750 con 0x7f69240a9d70 2024-03-29T10:42:32.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.529+0000 7f691a7fc640 1 -- 172.21.15.94:0/410578855 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f692412e750 con 0x7f69240a9d70 2024-03-29T10:42:32.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.529+0000 7f691a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:32.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.529+0000 7f691a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:32.513 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:32.513 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:32.516 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.532+0000 7f690ffff640 10 librados: watch_flush enter 2024-03-29T10:42:32.516 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.532+0000 7f690ffff640 10 librados: watch_flush exit 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.532+0000 7f690ffff640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.532+0000 7f690ffff640 10 client.4358.objecter successfully canceled tick 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.532+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 >> v1:172.21.15.94:6816/3829934808 conn(0x7f690806e390 legacy=0x7f6908070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 10 monclient: shutdown 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 >> v1:172.21.15.94:6790/0 conn(0x7f69240a9d70 legacy=0x7f6924142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 shutdown_connections 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 >> 172.21.15.94:0/410578855 conn(0x7f6924066a00 msgr2=0x7f69240b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 shutdown_connections 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 1 -- 172.21.15.94:0/410578855 wait complete. 2024-03-29T10:42:32.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:32.533+0000 7f690ffff640 1 librados: shutdown 2024-03-29T10:42:32.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:39.340 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.502+0000 7f4ff4123640 10 monclient: get_monmap_and_config 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.502+0000 7f4ff4123640 10 monclient: build_initial_monmap 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.502+0000 7f4ff4123640 10 monclient: monmap: 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:39.504016+0000 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:39.504016+0000 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:39.486 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:39.487 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:39.487 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:39.487 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:39.487 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:39.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.503+0000 7f4ff4123640 1 Processor -- start 2024-03-29T10:42:39.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.503+0000 7f4ff4123640 1 -- start start 2024-03-29T10:42:39.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.503+0000 7f4ff4123640 10 monclient: init 2024-03-29T10:42:39.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.504+0000 7f4ff4123640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:39.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.504+0000 7f4ff4123640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:39.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.504+0000 7f4ff4123640 10 monclient(hunting): picked mon.noname-b con 0x7f4fec1457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:39.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.504+0000 7f4ff1ebf640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f4fec1457f0 0x7f4fec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:42846/0 (socket says 172.21.15.94:42846) 2024-03-29T10:42:39.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff1ebf640 1 -- 172.21.15.94:0/3057636627 learned_addr learned my addr 172.21.15.94:0/3057636627 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:39.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff4123640 10 monclient(hunting): picked mon.noname-a con 0x7f4fec13f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff4123640 10 monclient(hunting): picked mon.noname-c con 0x7f4fec0a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0aaef0 con 0x7f4fec13f650 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec145bd0 con 0x7f4fec0a9d70 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec145eb0 con 0x7f4fec1457f0 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff4123640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff4123640 10 monclient(hunting): authenticate will time out at 1010.774457s 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178385967 0 0) 0x7f4fec145eb0 con 0x7f4fec1457f0 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8003520 con 0x7f4fec1457f0 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570515503 0 0) 0x7f4fd8003520 con 0x7f4fec1457f0 2024-03-29T10:42:39.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.505+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 >> v1:172.21.15.173:6789/0 conn(0x7f4fec0a9d70 legacy=0x7f4fec14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 >> v1:172.21.15.94:6789/0 conn(0x7f4fec13f650 legacy=0x7f4fec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 1 monclient: found mon.noname-b 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec146340 con 0x7f4fec1457f0 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: _finish_auth 0 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:39.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff4123640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff4123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4fe4001a10 con 0x7f4fec1457f0 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: dump: 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:39.494 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff4123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe4001ca0 con 0x7f4fec1457f0 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff4123640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff4123640 4 set_mon_vals no callback set 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.508+0000 7f4ff0ebd640 1 -- 172.21.15.94:0/3057636627 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4fe4002030 con 0x7f4fec1457f0 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.509+0000 7f4ff0ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.509+0000 7f4ff0ebd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.509+0000 7f4ff0ebd640 10 monclient: dump: 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:39.495 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.496 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.496 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:39.496 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:39.496 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:39.496 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:39.496 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:39.496 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:39.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.509+0000 7f4ff4123640 10 monclient: shutdown 2024-03-29T10:42:39.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.509+0000 7f4ff4123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:39.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.509+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 >> v1:172.21.15.94:6790/0 conn(0x7f4fec1457f0 legacy=0x7f4fec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:39.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.510+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 shutdown_connections 2024-03-29T10:42:39.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.510+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 >> 172.21.15.94:0/3057636627 conn(0x7f4fec066a00 msgr2=0x7f4fec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:39.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.510+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 shutdown_connections 2024-03-29T10:42:39.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.511+0000 7f4ff4123640 1 -- 172.21.15.94:0/3057636627 wait complete. 2024-03-29T10:42:39.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.513+0000 7f4ff4123640 10 monclient: build_initial_monmap 2024-03-29T10:42:39.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.513+0000 7f4ff4123640 10 monclient: monmap: 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.513+0000 7f4ff4123640 1 librados: starting msgr at 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.513+0000 7f4ff4123640 1 librados: starting objecter 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 1 Processor -- start 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 1 -- start start 2024-03-29T10:42:39.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 1 librados: setting wanted keys 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 1 librados: calling monclient init 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 10 monclient: init 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 10 monclient(hunting): picked mon.noname-a con 0x7f4fec0a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.514+0000 7f4ff4123640 10 monclient(hunting): picked mon.noname-c con 0x7f4fec13f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff4123640 10 monclient(hunting): picked mon.noname-b con 0x7f4fec1457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff4123640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0ad120 con 0x7f4fec0a9d70 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff4123640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0ad400 con 0x7f4fec1457f0 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff4123640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0ac1d0 con 0x7f4fec13f650 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff4123640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff4123640 10 monclient(hunting): authenticate will time out at 1010.784139s 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff16be640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f4fec13f650 0x7f4fec138070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:45554/0 (socket says 172.21.15.94:45554) 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.515+0000 7f4ff16be640 1 -- 172.21.15.94:0/2552196249 learned_addr learned my addr 172.21.15.94:0/2552196249 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:39.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 10 client.?.objecter ms_handle_connect 0x7f4fec13f650 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644219528 0 0) 0x7f4fec0ac1d0 con 0x7f4fec13f650 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd0004eb0 con 0x7f4fec13f650 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3712425126 0 0) 0x7f4fd0004eb0 con 0x7f4fec13f650 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fec0ac1d0 con 0x7f4fec13f650 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4fe80019d0 con 0x7f4fec13f650 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.516+0000 7f4fe27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.517+0000 7f4fe27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.517+0000 7f4fe27fc640 10 monclient(hunting): dump: 2024-03-29T10:42:39.505 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.517+0000 7f4fe27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.517+0000 7f4fe27fc640 10 client.?.objecter ms_handle_connect 0x7f4fec1457f0 2024-03-29T10:42:39.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.517+0000 7f4fe27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:39.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.519+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3129349999 0 0) 0x7f4fec0ac1d0 con 0x7f4fec13f650 2024-03-29T10:42:39.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.519+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 >> v1:172.21.15.173:6789/0 conn(0x7f4fec1457f0 legacy=0x7f4fec1e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:39.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.519+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 >> v1:172.21.15.94:6789/0 conn(0x7f4fec0a9d70 legacy=0x7f4fec140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.519+0000 7f4fe27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.519+0000 7f4fe27fc640 1 monclient: found mon.c 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.519+0000 7f4fe27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4fe27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0ac4e0 con 0x7f4fec13f650 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4fe27fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4fe27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4fe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4ff4123640 5 monclient: authenticate success, global_id 4364 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4ff4123640 10 monclient: _renew_subs 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.520+0000 7f4ff4123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec1429e0 con 0x7f4fec13f650 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4ff4123640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4ff4123640 10 monclient: _renew_subs 2024-03-29T10:42:39.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4ff4123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec140a30 con 0x7f4fec13f650 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4ff4123640 1 librados: init done 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe8001cd0 con 0x7f4fec13f650 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4fe27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4fe8003a60 con 0x7f4fec13f650 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4fe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4fe27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.521+0000 7f4fe27fc640 10 monclient: dump: 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:39.511 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.522+0000 7f4fe3fff640 4 set_mon_vals no callback set 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.523+0000 7f4ff4123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.523+0000 7f4ff4123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.523+0000 7f4ff4123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.523+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd4005080 con 0x7f4fec13f650 2024-03-29T10:42:39.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.524+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f4fe8003bc0 con 0x7f4fec13f650 2024-03-29T10:42:39.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.527+0000 7f4ff1ebf640 10 monclient: get_auth_request con 0x7f4fd006dc00 auth_method 0 2024-03-29T10:42:39.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.528+0000 7f4fe27fc640 10 client.4364.objecter ms_handle_connect 0x7f4fd006dc00 2024-03-29T10:42:39.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.528+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f4fe80a6b50 con 0x7f4fec13f650 2024-03-29T10:42:39.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.528+0000 7f4fe27fc640 10 client.4364.objecter ms_dispatch 0x7f4fec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:39.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.528+0000 7f4fe27fc640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:39.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.528+0000 7f4fe27fc640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:39.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.529+0000 7f4fe27fc640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T10:42:39.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.531+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fe8072260 con 0x7f4fec13f650 2024-03-29T10:42:39.516 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.532+0000 7f4fe27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:39.516 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.532+0000 7f4fe27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:39.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.858+0000 7f4ff4123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:39.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.858+0000 7f4ff4123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:39.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.858+0000 7f4ff4123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:39.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.858+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fd4005330 con 0x7f4fec13f650 2024-03-29T10:42:39.843 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:39.843 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:39.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.859+0000 7f4fe27fc640 1 -- 172.21.15.94:0/2552196249 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f4fe8074360 con 0x7f4fec13f650 2024-03-29T10:42:39.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.859+0000 7f4fe27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:39.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.859+0000 7f4fe27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:39.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.863+0000 7f4ff4123640 10 librados: watch_flush enter 2024-03-29T10:42:39.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.863+0000 7f4ff4123640 10 librados: watch_flush exit 2024-03-29T10:42:39.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.863+0000 7f4ff4123640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T10:42:39.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.863+0000 7f4ff4123640 10 client.4364.objecter successfully canceled tick 2024-03-29T10:42:39.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.864+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 >> v1:172.21.15.94:6816/3829934808 conn(0x7f4fd006dc00 legacy=0x7f4fd00700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:39.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.864+0000 7f4ff4123640 10 monclient: shutdown 2024-03-29T10:42:39.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.864+0000 7f4ff4123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:39.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.864+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 >> v1:172.21.15.94:6790/0 conn(0x7f4fec13f650 legacy=0x7f4fec138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:39.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.864+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 shutdown_connections 2024-03-29T10:42:39.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.864+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 >> 172.21.15.94:0/2552196249 conn(0x7f4fec066a00 msgr2=0x7f4fec0b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:39.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.864+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 shutdown_connections 2024-03-29T10:42:39.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.865+0000 7f4ff4123640 1 -- 172.21.15.94:0/2552196249 wait complete. 2024-03-29T10:42:39.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:39.865+0000 7f4ff4123640 1 librados: shutdown 2024-03-29T10:42:39.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:46.672 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.838+0000 7feb8957b640 10 monclient: get_monmap_and_config 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.838+0000 7feb8957b640 10 monclient: build_initial_monmap 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.838+0000 7feb8957b640 10 monclient: monmap: 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:46.839742+0000 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:46.839742+0000 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:46.822 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:46.823 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.841+0000 7feb8957b640 1 Processor -- start 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.841+0000 7feb8957b640 1 -- start start 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.841+0000 7feb8957b640 10 monclient: init 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 10 monclient(hunting): picked mon.noname-b con 0x7feb7c006e30 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 10 monclient(hunting): picked mon.noname-a con 0x7feb7c035e00 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c0c9670 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c035970 con 0x7feb7c035e00 2024-03-29T10:42:46.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c036280 con 0x7feb7c0c9670 2024-03-29T10:42:46.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c035640 con 0x7feb7c006e30 2024-03-29T10:42:46.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:46.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.842+0000 7feb8957b640 10 monclient(hunting): authenticate will time out at 1018.110751s 2024-03-29T10:42:46.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.845+0000 7feb837fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7feb7c0c9670 0x7feb7c0d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:50843/0 (socket says 172.21.15.94:50843) 2024-03-29T10:42:46.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.845+0000 7feb837fe640 1 -- 172.21.15.94:0/3158213773 learned_addr learned my addr 172.21.15.94:0/3158213773 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:46.833 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.849+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974824772 0 0) 0x7feb7c036280 con 0x7feb7c0c9670 2024-03-29T10:42:46.833 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.849+0000 7feb81ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:46.833 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.849+0000 7feb81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:46.833 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.849+0000 7feb81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:46.833 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.849+0000 7feb81ffb640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:42:46.833 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.849+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb6c003520 con 0x7feb7c0c9670 2024-03-29T10:42:46.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164339694 0 0) 0x7feb7c035970 con 0x7feb7c035e00 2024-03-29T10:42:46.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb6c003840 con 0x7feb7c035e00 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 599359630 0 0) 0x7feb6c003520 con 0x7feb7c0c9670 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 >> v1:172.21.15.94:6790/0 conn(0x7feb7c006e30 legacy=0x7feb7c0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 >> v1:172.21.15.94:6789/0 conn(0x7feb7c035e00 legacy=0x7feb7c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 1 monclient: found mon.noname-c 2024-03-29T10:42:46.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:46.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:42:46.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c034e20 con 0x7feb7c0c9670 2024-03-29T10:42:46.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:46.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:46.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.850+0000 7feb81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:46.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.851+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7feb8404f730 con 0x7feb7c0c9670 2024-03-29T10:42:46.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.851+0000 7feb81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:46.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.851+0000 7feb81ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:46.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.851+0000 7feb81ffb640 10 monclient: dump: 2024-03-29T10:42:46.837 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:46.837 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:46.837 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.837 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.837 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:46.838 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:46.838 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:46.838 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:46.838 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:46.838 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:46.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.851+0000 7feb8957b640 5 monclient: authenticate success, global_id 4357 2024-03-29T10:42:46.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.851+0000 7feb8957b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.852+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb84075f10 con 0x7feb7c0c9670 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.852+0000 7feb81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.852+0000 7feb8957b640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.852+0000 7feb8957b640 4 set_mon_vals no callback set 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.853+0000 7feb81ffb640 1 -- 172.21.15.94:0/3158213773 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7feb840758b0 con 0x7feb7c0c9670 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.853+0000 7feb81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.853+0000 7feb81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.853+0000 7feb81ffb640 10 monclient: dump: 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:46.841 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.853+0000 7feb8957b640 10 monclient: shutdown 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.853+0000 7feb8957b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.853+0000 7feb8957b640 1 -- 172.21.15.94:0/3158213773 >> v1:172.21.15.173:6789/0 conn(0x7feb7c0c9670 legacy=0x7feb7c0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.854+0000 7feb8957b640 1 -- 172.21.15.94:0/3158213773 shutdown_connections 2024-03-29T10:42:46.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.854+0000 7feb8957b640 1 -- 172.21.15.94:0/3158213773 >> 172.21.15.94:0/3158213773 conn(0x7feb7c005c00 msgr2=0x7feb7c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.855+0000 7feb8957b640 1 -- 172.21.15.94:0/3158213773 shutdown_connections 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.855+0000 7feb8957b640 1 -- 172.21.15.94:0/3158213773 wait complete. 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.859+0000 7feb8957b640 10 monclient: build_initial_monmap 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.859+0000 7feb8957b640 10 monclient: monmap: 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:46.843 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:46.844 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:46.844 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:46.844 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:46.846 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.860+0000 7feb8957b640 1 librados: starting msgr at 2024-03-29T10:42:46.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.860+0000 7feb8957b640 1 librados: starting objecter 2024-03-29T10:42:46.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 1 Processor -- start 2024-03-29T10:42:46.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 1 -- start start 2024-03-29T10:42:46.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 1 librados: setting wanted keys 2024-03-29T10:42:46.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 1 librados: calling monclient init 2024-03-29T10:42:46.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient: init 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c006e30 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient(hunting): picked mon.noname-b con 0x7feb7c035e00 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient(hunting): picked mon.noname-a con 0x7feb7c0c9670 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c02d150 con 0x7feb7c0c9670 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c02f1b0 con 0x7feb7c035e00 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c02f490 con 0x7feb7c006e30 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:46.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.861+0000 7feb8957b640 10 monclient(hunting): authenticate will time out at 1018.130048s 2024-03-29T10:42:46.857 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.867+0000 7feb82ffd640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7feb7c006e30 0x7feb7c02dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:51507/0 (socket says 172.21.15.94:51507) 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.867+0000 7feb82ffd640 1 -- 172.21.15.94:0/518919165 learned_addr learned my addr 172.21.15.94:0/518919165 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.867+0000 7feb6b7fe640 10 client.?.objecter ms_handle_connect 0x7feb7c006e30 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.867+0000 7feb6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733810390 0 0) 0x7feb7c02f490 con 0x7feb7c006e30 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70004eb0 con 0x7feb7c006e30 2024-03-29T10:42:46.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 client.?.objecter ms_handle_connect 0x7feb7c0c9670 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165973778 0 0) 0x7feb7c02d150 con 0x7feb7c0c9670 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb700051d0 con 0x7feb7c0c9670 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1039930162 0 0) 0x7feb70004eb0 con 0x7feb7c006e30 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb70005640 con 0x7feb7c006e30 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7feb74001920 con 0x7feb7c006e30 2024-03-29T10:42:46.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): dump: 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.868+0000 7feb6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:46.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048906087 0 0) 0x7feb700051d0 con 0x7feb7c0c9670 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb70005ce0 con 0x7feb7c0c9670 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7feb8404f400 con 0x7feb7c0c9670 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 monclient(hunting): dump: 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:46.861 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 client.?.objecter ms_handle_connect 0x7feb7c035e00 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3757899233 0 0) 0x7feb70005640 con 0x7feb7c006e30 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 >> v1:172.21.15.173:6789/0 conn(0x7feb7c035e00 legacy=0x7feb7c15f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 >> v1:172.21.15.94:6789/0 conn(0x7feb7c0c9670 legacy=0x7feb7c16cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 monclient: found mon.c 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c02e5f0 con 0x7feb7c006e30 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:46.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:46.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:46.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 5 monclient: authenticate success, global_id 4367 2024-03-29T10:42:46.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 10 monclient: _renew_subs 2024-03-29T10:42:46.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:46.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c15fc20 con 0x7feb7c006e30 2024-03-29T10:42:46.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 10 monclient: _renew_subs 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c16d630 con 0x7feb7c006e30 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.869+0000 7feb8957b640 1 librados: init done 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.870+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb74001c40 con 0x7feb7c006e30 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.870+0000 7feb6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.870+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7feb74001920 con 0x7feb7c006e30 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.870+0000 7feb6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.870+0000 7feb6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.870+0000 7feb6b7fe640 10 monclient: dump: 2024-03-29T10:42:46.864 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.870+0000 7feb817fa640 4 set_mon_vals no callback set 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.871+0000 7feb8957b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.871+0000 7feb8957b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.871+0000 7feb8957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:46.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.871+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64005080 con 0x7feb7c006e30 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.872+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7feb740038a0 con 0x7feb7c006e30 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.872+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7feb740543e0 con 0x7feb7c006e30 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.872+0000 7feb6b7fe640 10 client.4367.objecter ms_dispatch 0x7feb84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.872+0000 7feb6b7fe640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.872+0000 7feb6b7fe640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.872+0000 7feb6b7fe640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.872+0000 7feb827fc640 10 monclient: get_auth_request con 0x7feb7006e860 auth_method 0 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.874+0000 7feb6b7fe640 10 client.4367.objecter ms_handle_connect 0x7feb7006e860 2024-03-29T10:42:46.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.881+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb74050680 con 0x7feb7c006e30 2024-03-29T10:42:46.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.881+0000 7feb6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:46.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:46.881+0000 7feb6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:47.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.202+0000 7feb8957b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:47.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.202+0000 7feb8957b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:47.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.202+0000 7feb8957b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:47.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.202+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb64005330 con 0x7feb7c006e30 2024-03-29T10:42:47.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.204+0000 7feb6b7fe640 1 -- 172.21.15.94:0/518919165 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7feb74050820 con 0x7feb7c006e30 2024-03-29T10:42:47.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.204+0000 7feb6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:47.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.204+0000 7feb6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:47.187 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:47.187 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:47.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.207+0000 7feb8957b640 10 librados: watch_flush enter 2024-03-29T10:42:47.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.209+0000 7feb8957b640 10 librados: watch_flush exit 2024-03-29T10:42:47.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.209+0000 7feb8957b640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T10:42:47.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.209+0000 7feb8957b640 10 client.4367.objecter successfully canceled tick 2024-03-29T10:42:47.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.209+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 >> v1:172.21.15.94:6816/3829934808 conn(0x7feb7006e860 legacy=0x7feb70070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:47.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.209+0000 7feb8957b640 10 monclient: shutdown 2024-03-29T10:42:47.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.209+0000 7feb8957b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:47.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.209+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 >> v1:172.21.15.94:6790/0 conn(0x7feb7c006e30 legacy=0x7feb7c02dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:47.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.211+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 shutdown_connections 2024-03-29T10:42:47.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.211+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 >> 172.21.15.94:0/518919165 conn(0x7feb7c005c00 msgr2=0x7feb7c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:47.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.211+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 shutdown_connections 2024-03-29T10:42:47.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.212+0000 7feb8957b640 1 -- 172.21.15.94:0/518919165 wait complete. 2024-03-29T10:42:47.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:47.212+0000 7feb8957b640 1 librados: shutdown 2024-03-29T10:42:47.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:54.020 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:54.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.197+0000 7f8726ed9640 10 monclient: get_monmap_and_config 2024-03-29T10:42:54.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.197+0000 7f8726ed9640 10 monclient: build_initial_monmap 2024-03-29T10:42:54.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.197+0000 7f8726ed9640 10 monclient: monmap: 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:42:54.198518+0000 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:42:54.198518+0000 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:54.181 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.198+0000 7f8726ed9640 1 Processor -- start 2024-03-29T10:42:54.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.199+0000 7f8726ed9640 1 -- start start 2024-03-29T10:42:54.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.199+0000 7f8726ed9640 10 monclient: init 2024-03-29T10:42:54.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.199+0000 7f8726ed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:54.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.199+0000 7f8726ed9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:54.182 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.199+0000 7f8726ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f87201422b0 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:54.186 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.203+0000 7f8724c75640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f87201422b0 0x7f872013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:47207/0 (socket says 172.21.15.94:47207) 2024-03-29T10:42:54.187 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.204+0000 7f871ffff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f8720142690 0x7f87200b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50983/0 (socket says 172.21.15.94:50983) 2024-03-29T10:42:54.188 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.204+0000 7f8726ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f8720142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:54.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.206+0000 7f8726ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f87200b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:54.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.206+0000 7f8726ed9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8720142c80 con 0x7f8720142690 2024-03-29T10:42:54.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.206+0000 7f8726ed9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87200aae40 con 0x7f87200b17d0 2024-03-29T10:42:54.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.206+0000 7f8726ed9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872013fbd0 con 0x7f87201422b0 2024-03-29T10:42:54.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.206+0000 7f8726ed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:54.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.206+0000 7f8726ed9640 10 monclient(hunting): authenticate will time out at 1025.474757s 2024-03-29T10:42:54.189 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.206+0000 7f871ffff640 1 -- 172.21.15.94:0/3999429579 learned_addr learned my addr 172.21.15.94:0/3999429579 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:54.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786660314 0 0) 0x7f8720142c80 con 0x7f8720142690 2024-03-29T10:42:54.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:54.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:54.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:54.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8710003520 con 0x7f8720142690 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249311367 0 0) 0x7f872013fbd0 con 0x7f87201422b0 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8720142c80 con 0x7f87201422b0 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 870368487 0 0) 0x7f8710003520 con 0x7f8720142690 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 >> v1:172.21.15.94:6790/0 conn(0x7f87201422b0 legacy=0x7f872013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:54.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.207+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 >> v1:172.21.15.173:6789/0 conn(0x7f87200b17d0 legacy=0x7f87200b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 1 monclient: found mon.noname-a 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8720151940 con 0x7f8720142690 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:54.193 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f870c001690 con 0x7f8720142690 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f871f7fe640 10 monclient: dump: 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:54.194 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:54.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f8726ed9640 5 monclient: authenticate success, global_id 4377 2024-03-29T10:42:54.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.210+0000 7f8726ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:54.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f870c001870 con 0x7f8720142690 2024-03-29T10:42:54.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f871f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:54.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f871f7fe640 1 -- 172.21.15.94:0/3999429579 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f870c001c00 con 0x7f8720142690 2024-03-29T10:42:54.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f871f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f871f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f871f7fe640 10 monclient: dump: 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f8726ed9640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:54.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.211+0000 7f8726ed9640 4 set_mon_vals no callback set 2024-03-29T10:42:54.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.212+0000 7f8726ed9640 10 monclient: shutdown 2024-03-29T10:42:54.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.212+0000 7f8726ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:54.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.212+0000 7f8726ed9640 1 -- 172.21.15.94:0/3999429579 >> v1:172.21.15.94:6789/0 conn(0x7f8720142690 legacy=0x7f87200b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:54.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.212+0000 7f8726ed9640 1 -- 172.21.15.94:0/3999429579 shutdown_connections 2024-03-29T10:42:54.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.212+0000 7f8726ed9640 1 -- 172.21.15.94:0/3999429579 >> 172.21.15.94:0/3999429579 conn(0x7f8720066a00 msgr2=0x7f8720066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:54.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.213+0000 7f8726ed9640 1 -- 172.21.15.94:0/3999429579 shutdown_connections 2024-03-29T10:42:54.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.213+0000 7f8726ed9640 1 -- 172.21.15.94:0/3999429579 wait complete. 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 10 monclient: build_initial_monmap 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 10 monclient: monmap: 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:54.199 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 1 librados: starting msgr at 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 1 librados: starting objecter 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 1 Processor -- start 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 1 -- start start 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 1 librados: setting wanted keys 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 1 librados: calling monclient init 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.214+0000 7f8726ed9640 10 monclient: init 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.215+0000 7f8726ed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.215+0000 7f8726ed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:54.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.215+0000 7f8726ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f87200b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.218+0000 7f8726ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f87201422b0 addr v1:172.21.15.173:6789/0 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.218+0000 7f8726ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f8720142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.218+0000 7f8726ed9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87200acc80 con 0x7f8720142690 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.218+0000 7f8726ed9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87200acf60 con 0x7f87201422b0 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.218+0000 7f8726ed9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8720131a00 con 0x7f87200b17d0 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.218+0000 7f8726ed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.218+0000 7f8726ed9640 10 monclient(hunting): authenticate will time out at 1025.486792s 2024-03-29T10:42:54.207 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f8724c75640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f87200b17d0 0x7f87200ab960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:47323/0 (socket says 172.21.15.94:47323) 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f8724c75640 1 -- 172.21.15.94:0/2845378372 learned_addr learned my addr 172.21.15.94:0/2845378372 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 10 client.?.objecter ms_handle_connect 0x7f87200b17d0 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794030997 0 0) 0x7f8720131a00 con 0x7f87200b17d0 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.221+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f870400d6c0 con 0x7f87200b17d0 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.222+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 131318082 0 0) 0x7f870400d6c0 con 0x7f87200b17d0 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.222+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8720131a00 con 0x7f87200b17d0 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.222+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f87140019f0 con 0x7f87200b17d0 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.222+0000 7f871d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.222+0000 7f871d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:54.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.222+0000 7f871d7fa640 10 monclient(hunting): dump: 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.222+0000 7f871d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 10 client.?.objecter ms_handle_connect 0x7f87201422b0 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:54.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3509879864 0 0) 0x7f8720131a00 con 0x7f87200b17d0 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 >> v1:172.21.15.173:6789/0 conn(0x7f87201422b0 legacy=0x7f872012e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 >> v1:172.21.15.94:6789/0 conn(0x7f8720142690 legacy=0x7f8720086540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 1 monclient: found mon.c 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8720131b90 con 0x7f87200b17d0 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:42:54.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.224+0000 7f871d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:54.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:42:54.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 10 monclient: _renew_subs 2024-03-29T10:42:54.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:54.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8720131d80 con 0x7f87200b17d0 2024-03-29T10:42:54.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 10 monclient: _renew_subs 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87200b7480 con 0x7f87200b17d0 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.225+0000 7f8726ed9640 1 librados: init done 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.226+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8714001b50 con 0x7f87200b17d0 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.226+0000 7f871d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.226+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8714004c30 con 0x7f87200b17d0 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.226+0000 7f871d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.226+0000 7f871d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.226+0000 7f871d7fa640 10 monclient: dump: 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:42:54.213 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:54.214 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:42:54.214 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:42:54.214 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:42:54.214 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:42:54.214 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:42:54.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.226+0000 7f871effd640 4 set_mon_vals no callback set 2024-03-29T10:42:54.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.228+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f871401ac00 con 0x7f87200b17d0 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f8714084c60 con 0x7f87200b17d0 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f871d7fa640 10 client.4373.objecter ms_dispatch 0x7f8720000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f871d7fa640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f871d7fa640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f871d7fa640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f8726ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f8726ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f8726ed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.229+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8700005080 con 0x7f87200b17d0 2024-03-29T10:42:54.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.231+0000 7f871ffff640 10 monclient: get_auth_request con 0x7f8704076420 auth_method 0 2024-03-29T10:42:54.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.231+0000 7f871d7fa640 10 client.4373.objecter ms_handle_connect 0x7f8704076420 2024-03-29T10:42:54.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.241+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8714050540 con 0x7f87200b17d0 2024-03-29T10:42:54.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.241+0000 7f871d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:54.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.241+0000 7f871d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:54.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.566+0000 7f8726ed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:54.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.566+0000 7f8726ed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:54.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.566+0000 7f8726ed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:42:54.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.566+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8700005330 con 0x7f87200b17d0 2024-03-29T10:42:54.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.570+0000 7f871d7fa640 1 -- 172.21.15.94:0/2845378372 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f8714052470 con 0x7f87200b17d0 2024-03-29T10:42:54.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.570+0000 7f871d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:54.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.570+0000 7f871d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:54.553 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:42:54.553 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 10 librados: watch_flush enter 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 10 librados: watch_flush exit 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 10 client.4373.objecter successfully canceled tick 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 >> v1:172.21.15.94:6816/3829934808 conn(0x7f8704076420 legacy=0x7f87040788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 10 monclient: shutdown 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.573+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 >> v1:172.21.15.94:6790/0 conn(0x7f87200b17d0 legacy=0x7f87200ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:54.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.574+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 shutdown_connections 2024-03-29T10:42:54.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.574+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 >> 172.21.15.94:0/2845378372 conn(0x7f8720066a00 msgr2=0x7f872013dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:54.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.574+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 shutdown_connections 2024-03-29T10:42:54.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.574+0000 7f8726ed9640 1 -- 172.21.15.94:0/2845378372 wait complete. 2024-03-29T10:42:54.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:42:54.574+0000 7f8726ed9640 1 librados: shutdown 2024-03-29T10:42:54.574 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:01.380 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.550+0000 7f5f471cf640 10 monclient: get_monmap_and_config 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.550+0000 7f5f471cf640 10 monclient: build_initial_monmap 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.550+0000 7f5f471cf640 10 monclient: monmap: 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:01.551666+0000 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:01.551666+0000 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:01.534 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:01.535 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:01.535 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.551+0000 7f5f471cf640 1 Processor -- start 2024-03-29T10:43:01.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.551+0000 7f5f471cf640 1 -- start start 2024-03-29T10:43:01.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.551+0000 7f5f471cf640 10 monclient: init 2024-03-29T10:43:01.535 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 10 monclient(hunting): picked mon.noname-b con 0x7f5f400af290 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 10 monclient(hunting): picked mon.noname-c con 0x7f5f400ae740 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 10 monclient(hunting): picked mon.noname-a con 0x7f5f400af670 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400b0980 con 0x7f5f400af670 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400b39f0 con 0x7f5f400ae740 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400aebd0 con 0x7f5f400af290 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:01.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.552+0000 7f5f471cf640 10 monclient(hunting): authenticate will time out at 1032.820931s 2024-03-29T10:43:01.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.555+0000 7f5f4576c640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5f400af670 0x7f5f40151e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50155/0 (socket says 172.21.15.94:50155) 2024-03-29T10:43:01.539 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.556+0000 7f5f4576c640 1 -- 172.21.15.94:0/122054100 learned_addr learned my addr 172.21.15.94:0/122054100 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978683547 0 0) 0x7f5f400aebd0 con 0x7f5f400af290 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f30003520 con 0x7f5f400af290 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519881398 0 0) 0x7f5f400b39f0 con 0x7f5f400ae740 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f400aebd0 con 0x7f5f400ae740 2024-03-29T10:43:01.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 535600591 0 0) 0x7f5f400aebd0 con 0x7f5f400ae740 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 >> v1:172.21.15.94:6790/0 conn(0x7f5f400af290 legacy=0x7f5f400b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 >> v1:172.21.15.94:6789/0 conn(0x7f5f400af670 legacy=0x7f5f40151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400b4b40 con 0x7f5f400ae740 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.560+0000 7f5f3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5f2c001670 con 0x7f5f400ae740 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:01.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 10 monclient: dump: 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f2c0018c0 con 0x7f5f400ae740 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:01.547 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 1 -- 172.21.15.94:0/122054100 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5f2c001c50 con 0x7f5f400ae740 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.561+0000 7f5f3f7fe640 10 monclient: dump: 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:01.548 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.549 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.562+0000 7f5f471cf640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.562+0000 7f5f471cf640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.562+0000 7f5f471cf640 4 set_mon_vals no callback set 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.563+0000 7f5f471cf640 10 monclient: shutdown 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.563+0000 7f5f471cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.563+0000 7f5f471cf640 1 -- 172.21.15.94:0/122054100 >> v1:172.21.15.173:6789/0 conn(0x7f5f400ae740 legacy=0x7f5f401439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.563+0000 7f5f471cf640 1 -- 172.21.15.94:0/122054100 shutdown_connections 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.563+0000 7f5f471cf640 1 -- 172.21.15.94:0/122054100 >> 172.21.15.94:0/122054100 conn(0x7f5f40066a00 msgr2=0x7f5f40066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:01.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.564+0000 7f5f471cf640 1 -- 172.21.15.94:0/122054100 shutdown_connections 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.566+0000 7f5f471cf640 1 -- 172.21.15.94:0/122054100 wait complete. 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 10 monclient: build_initial_monmap 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 10 monclient: monmap: 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:01.552 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 1 librados: starting msgr at 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 1 librados: starting objecter 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 1 Processor -- start 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 1 -- start start 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 1 librados: setting wanted keys 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 1 librados: calling monclient init 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.567+0000 7f5f471cf640 10 monclient: init 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 10 monclient(hunting): picked mon.noname-b con 0x7f5f400af290 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 10 monclient(hunting): picked mon.noname-a con 0x7f5f400af670 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 10 monclient(hunting): picked mon.noname-c con 0x7f5f401454e0 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f401472c0 con 0x7f5f400af670 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f401db4b0 con 0x7f5f400af290 2024-03-29T10:43:01.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f401433f0 con 0x7f5f401454e0 2024-03-29T10:43:01.554 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:01.554 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.568+0000 7f5f471cf640 10 monclient(hunting): authenticate will time out at 1032.836877s 2024-03-29T10:43:01.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.573+0000 7f5f44f6b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f5f400af290 0x7f5f40144dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:35785/0 (socket says 172.21.15.94:35785) 2024-03-29T10:43:01.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.573+0000 7f5f44f6b640 1 -- 172.21.15.94:0/1670094155 learned_addr learned my addr 172.21.15.94:0/1670094155 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:01.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 client.?.objecter ms_handle_connect 0x7f5f400af290 2024-03-29T10:43:01.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:01.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 client.?.objecter ms_handle_connect 0x7f5f401454e0 2024-03-29T10:43:01.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:01.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 client.?.objecter ms_handle_connect 0x7f5f400af670 2024-03-29T10:43:01.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822882680 0 0) 0x7f5f401db4b0 con 0x7f5f400af290 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f24004eb0 con 0x7f5f400af290 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902382805 0 0) 0x7f5f401472c0 con 0x7f5f400af670 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f401db4b0 con 0x7f5f400af670 2024-03-29T10:43:01.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31117033 0 0) 0x7f5f401433f0 con 0x7f5f401454e0 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f401472c0 con 0x7f5f401454e0 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2472003782 0 0) 0x7f5f401db4b0 con 0x7f5f400af670 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f401433f0 con 0x7f5f400af670 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5f2c001630 con 0x7f5f400af670 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.567 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2674846277 0 0) 0x7f5f401472c0 con 0x7f5f401454e0 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f401db4b0 con 0x7f5f401454e0 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5f38001940 con 0x7f5f401454e0 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.575+0000 7f5f3d7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:01.568 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180047560 0 0) 0x7f5f24004eb0 con 0x7f5f400af290 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f401472c0 con 0x7f5f400af290 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5f34001990 con 0x7f5f400af290 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:01.569 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2965090566 0 0) 0x7f5f401433f0 con 0x7f5f400af670 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 >> v1:172.21.15.94:6790/0 conn(0x7f5f401454e0 legacy=0x7f5f401e6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 >> v1:172.21.15.173:6789/0 conn(0x7f5f400af290 legacy=0x7f5f40144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 monclient: found mon.a 2024-03-29T10:43:01.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f40146da0 con 0x7f5f400af670 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 5 monclient: authenticate success, global_id 4380 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 10 monclient: _renew_subs 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40146f90 con 0x7f5f400af670 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 10 monclient: _renew_subs 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f40144ad0 con 0x7f5f400af670 2024-03-29T10:43:01.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f471cf640 1 librados: init done 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f2c0038a0 con 0x7f5f400af670 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5f2c003bb0 con 0x7f5f400af670 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3d7fa640 10 monclient: dump: 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:01.572 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.576+0000 7f5f3effd640 4 set_mon_vals no callback set 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.578+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f5f2c019b80 con 0x7f5f400af670 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.579+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f5f2c0837d0 con 0x7f5f400af670 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.579+0000 7f5f3d7fa640 10 client.4380.objecter ms_dispatch 0x7f5f40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.579+0000 7f5f3d7fa640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.579+0000 7f5f3d7fa640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.579+0000 7f5f44f6b640 10 monclient: get_auth_request con 0x7f5f2406e0c0 auth_method 0 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.579+0000 7f5f3d7fa640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.580+0000 7f5f3d7fa640 10 client.4380.objecter ms_handle_connect 0x7f5f2406e0c0 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.581+0000 7f5f471cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.581+0000 7f5f471cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.581+0000 7f5f471cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:01.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.581+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f20005080 con 0x7f5f400af670 2024-03-29T10:43:01.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.593+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f2c04f0b0 con 0x7f5f400af670 2024-03-29T10:43:01.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.593+0000 7f5f3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:01.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.593+0000 7f5f3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:01.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.923+0000 7f5f471cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:01.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.923+0000 7f5f471cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:01.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.923+0000 7f5f471cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:01.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.923+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f20005330 con 0x7f5f400af670 2024-03-29T10:43:01.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.932+0000 7f5f3d7fa640 1 -- 172.21.15.94:0/1670094155 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f5f2c050fe0 con 0x7f5f400af670 2024-03-29T10:43:01.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.932+0000 7f5f3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:01.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.932+0000 7f5f3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:01.917 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:01.918 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 10 librados: watch_flush enter 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 10 librados: watch_flush exit 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 10 client.4380.objecter successfully canceled tick 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 >> v1:172.21.15.94:6816/3829934808 conn(0x7f5f2406e0c0 legacy=0x7f5f24070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 10 monclient: shutdown 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 >> v1:172.21.15.94:6789/0 conn(0x7f5f400af670 legacy=0x7f5f401db9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:01.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 shutdown_connections 2024-03-29T10:43:01.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.939+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 >> 172.21.15.94:0/1670094155 conn(0x7f5f40066a00 msgr2=0x7f5f4013fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:01.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.941+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 shutdown_connections 2024-03-29T10:43:01.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.941+0000 7f5f471cf640 1 -- 172.21.15.94:0/1670094155 wait complete. 2024-03-29T10:43:01.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:01.941+0000 7f5f471cf640 1 librados: shutdown 2024-03-29T10:43:01.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:08.750 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:08.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.915+0000 7f89aa40c640 10 monclient: get_monmap_and_config 2024-03-29T10:43:08.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.915+0000 7f89aa40c640 10 monclient: build_initial_monmap 2024-03-29T10:43:08.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.915+0000 7f89aa40c640 10 monclient: monmap: 2024-03-29T10:43:08.901 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:08.901 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:08.901 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:08.916122+0000 2024-03-29T10:43:08.901 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:08.916122+0000 2024-03-29T10:43:08.902 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:08.902 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:08.902 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:08.902 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:08.902 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:08.902 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:08.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.918+0000 7f89aa40c640 1 Processor -- start 2024-03-29T10:43:08.904 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.918+0000 7f89aa40c640 1 -- start start 2024-03-29T10:43:08.904 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.918+0000 7f89aa40c640 10 monclient: init 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 10 monclient(hunting): picked mon.noname-c con 0x7f89a40a9ac0 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 10 monclient(hunting): picked mon.noname-b con 0x7f89a413ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 10 monclient(hunting): picked mon.noname-a con 0x7f89a40af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a40afd10 con 0x7f89a40af900 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a40af6f0 con 0x7f89a40a9ac0 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a40aac80 con 0x7f89a413ee90 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:08.905 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.919+0000 7f89aa40c640 10 monclient(hunting): authenticate will time out at 1040.187755s 2024-03-29T10:43:08.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.923+0000 7f89a89a9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f89a40af900 0x7f89a4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49025/0 (socket says 172.21.15.94:49025) 2024-03-29T10:43:08.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.923+0000 7f89a89a9640 1 -- 172.21.15.94:0/412586202 learned_addr learned my addr 172.21.15.94:0/412586202 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:08.907 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.924+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052390416 0 0) 0x7f89a40afd10 con 0x7f89a40af900 2024-03-29T10:43:08.907 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.924+0000 7f89a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:08.907 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.924+0000 7f89a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:08.907 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.924+0000 7f89a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:08.907 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.924+0000 7f89a2ffd640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:43:08.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.924+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8988003520 con 0x7f89a40af900 2024-03-29T10:43:08.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.925+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020888505 0 0) 0x7f8988003520 con 0x7f89a40af900 2024-03-29T10:43:08.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.925+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 >> v1:172.21.15.94:6790/0 conn(0x7f89a413ee90 legacy=0x7f89a413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:08.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.925+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 >> v1:172.21.15.173:6789/0 conn(0x7f89a40a9ac0 legacy=0x7f89a40adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 1 monclient: found mon.noname-a 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a40aaf60 con 0x7f89a40af900 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89aa40c640 5 monclient: authenticate success, global_id 4383 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89aa40c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f8990001690 con 0x7f89a40af900 2024-03-29T10:43:08.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: dump: 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:08.912 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89aa40c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 1 -- 172.21.15.94:0/412586202 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8990001920 con 0x7f89a40af900 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.926+0000 7f89a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.927+0000 7f89aa40c640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.927+0000 7f89aa40c640 4 set_mon_vals no callback set 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.927+0000 7f89aa40c640 10 monclient: shutdown 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.927+0000 7f89aa40c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.927+0000 7f89aa40c640 1 -- 172.21.15.94:0/412586202 >> v1:172.21.15.94:6789/0 conn(0x7f89a40af900 legacy=0x7f89a4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.928+0000 7f89aa40c640 1 -- 172.21.15.94:0/412586202 shutdown_connections 2024-03-29T10:43:08.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.928+0000 7f89aa40c640 1 -- 172.21.15.94:0/412586202 >> 172.21.15.94:0/412586202 conn(0x7f89a40b35e0 msgr2=0x7f89a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.928+0000 7f89aa40c640 1 -- 172.21.15.94:0/412586202 shutdown_connections 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.928+0000 7f89aa40c640 1 -- 172.21.15.94:0/412586202 wait complete. 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.929+0000 7f89aa40c640 10 monclient: build_initial_monmap 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.929+0000 7f89aa40c640 10 monclient: monmap: 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:08.914 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.930+0000 7f89aa40c640 1 librados: starting msgr at 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.930+0000 7f89aa40c640 1 librados: starting objecter 2024-03-29T10:43:08.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.931+0000 7f89aa40c640 1 Processor -- start 2024-03-29T10:43:08.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.933+0000 7f89aa40c640 1 -- start start 2024-03-29T10:43:08.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.933+0000 7f89aa40c640 1 librados: setting wanted keys 2024-03-29T10:43:08.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.933+0000 7f89aa40c640 1 librados: calling monclient init 2024-03-29T10:43:08.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.933+0000 7f89aa40c640 10 monclient: init 2024-03-29T10:43:08.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.934+0000 7f89aa40c640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:08.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.934+0000 7f89aa40c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:08.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.934+0000 7f89aa40c640 10 monclient(hunting): picked mon.noname-b con 0x7f89a40a9ac0 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:08.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.938+0000 7f89aa40c640 10 monclient(hunting): picked mon.noname-c con 0x7f89a40af900 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:08.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.938+0000 7f89aa40c640 10 monclient(hunting): picked mon.noname-a con 0x7f89a413ee90 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:08.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.938+0000 7f89aa40c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a4141390 con 0x7f89a413ee90 2024-03-29T10:43:08.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.938+0000 7f89aa40c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a406f570 con 0x7f89a40a9ac0 2024-03-29T10:43:08.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.938+0000 7f89aa40c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a406f850 con 0x7f89a40af900 2024-03-29T10:43:08.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.938+0000 7f89aa40c640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:08.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.938+0000 7f89aa40c640 10 monclient(hunting): authenticate will time out at 1040.206243s 2024-03-29T10:43:08.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.939+0000 7f89a3fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f89a40a9ac0 0x7f89a4141870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:54483/0 (socket says 172.21.15.94:54483) 2024-03-29T10:43:08.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.940+0000 7f89a3fff640 1 -- 172.21.15.94:0/2079590384 learned_addr learned my addr 172.21.15.94:0/2079590384 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:08.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.941+0000 7f89a0ff9640 10 client.?.objecter ms_handle_connect 0x7f89a40a9ac0 2024-03-29T10:43:08.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.941+0000 7f89a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:08.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 10 client.?.objecter ms_handle_connect 0x7f89a40af900 2024-03-29T10:43:08.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219426439 0 0) 0x7f89a406f850 con 0x7f89a40af900 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.944+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f898c004eb0 con 0x7f89a40af900 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622827598 0 0) 0x7f89a406f570 con 0x7f89a40a9ac0 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a406f850 con 0x7f89a40a9ac0 2024-03-29T10:43:08.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 client.?.objecter ms_handle_connect 0x7f89a413ee90 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768010268 0 0) 0x7f89a4141390 con 0x7f89a413ee90 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a406f570 con 0x7f89a413ee90 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.945+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 194083953 0 0) 0x7f89a406f850 con 0x7f89a40a9ac0 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89a4141390 con 0x7f89a40a9ac0 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f899c001b30 con 0x7f89a40a9ac0 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient(hunting): dump: 2024-03-29T10:43:08.933 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3096858946 0 0) 0x7f89a4141390 con 0x7f89a40a9ac0 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 >> v1:172.21.15.94:6790/0 conn(0x7f89a40af900 legacy=0x7f89a4141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 >> v1:172.21.15.94:6789/0 conn(0x7f89a413ee90 legacy=0x7f89a41e1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:08.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 monclient: found mon.b 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4138db0 con 0x7f89a40a9ac0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89aa40c640 5 monclient: authenticate success, global_id 4366 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89aa40c640 10 monclient: _renew_subs 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89aa40c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a4139080 con 0x7f89a40a9ac0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f899c002040 con 0x7f89a40a9ac0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f899c004e20 con 0x7f89a40a9ac0 2024-03-29T10:43:08.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.946+0000 7f89a0ff9640 10 monclient: dump: 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.947+0000 7f89aa40c640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.947+0000 7f89aa40c640 10 monclient: _renew_subs 2024-03-29T10:43:08.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.947+0000 7f89aa40c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:08.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.947+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a413def0 con 0x7f89a40a9ac0 2024-03-29T10:43:08.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.947+0000 7f89aa40c640 1 librados: init done 2024-03-29T10:43:08.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.947+0000 7f89a27fc640 4 set_mon_vals no callback set 2024-03-29T10:43:08.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.949+0000 7f89aa40c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:08.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.950+0000 7f89aa40c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:08.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.950+0000 7f89aa40c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:08.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.950+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8980005080 con 0x7f89a40a9ac0 2024-03-29T10:43:08.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.952+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f899c01adf0 con 0x7f89a40a9ac0 2024-03-29T10:43:08.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.953+0000 7f89a37fe640 10 monclient: get_auth_request con 0x7f898c06ddd0 auth_method 0 2024-03-29T10:43:08.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.953+0000 7f89a0ff9640 10 client.4366.objecter ms_handle_connect 0x7f898c06ddd0 2024-03-29T10:43:08.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.953+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f899c084d00 con 0x7f89a40a9ac0 2024-03-29T10:43:08.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.953+0000 7f89a0ff9640 10 client.4366.objecter ms_dispatch 0x7f89a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:08.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.953+0000 7f89a0ff9640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:08.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.953+0000 7f89a0ff9640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:08.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.953+0000 7f89a0ff9640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T10:43:08.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.959+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f899c0505e0 con 0x7f89a40a9ac0 2024-03-29T10:43:08.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.959+0000 7f89a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:08.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:08.959+0000 7f89a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:09.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.281+0000 7f89aa40c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:09.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.281+0000 7f89aa40c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:09.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.281+0000 7f89aa40c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:09.265 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.281+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8980005330 con 0x7f89a40a9ac0 2024-03-29T10:43:09.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.283+0000 7f89a0ff9640 1 -- 172.21.15.94:0/2079590384 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f899c052510 con 0x7f89a40a9ac0 2024-03-29T10:43:09.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.283+0000 7f89a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:09.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.283+0000 7f89a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:09.268 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:09.268 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:09.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.288+0000 7f89aa40c640 10 librados: watch_flush enter 2024-03-29T10:43:09.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.288+0000 7f89aa40c640 10 librados: watch_flush exit 2024-03-29T10:43:09.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.288+0000 7f89aa40c640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T10:43:09.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.288+0000 7f89aa40c640 10 client.4366.objecter successfully canceled tick 2024-03-29T10:43:09.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.292+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 >> v1:172.21.15.94:6816/3829934808 conn(0x7f898c06ddd0 legacy=0x7f898c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:09.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.292+0000 7f89aa40c640 10 monclient: shutdown 2024-03-29T10:43:09.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.292+0000 7f89aa40c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:09.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.292+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 >> v1:172.21.15.173:6789/0 conn(0x7f89a40a9ac0 legacy=0x7f89a4141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:09.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.294+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 shutdown_connections 2024-03-29T10:43:09.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.294+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 >> 172.21.15.94:0/2079590384 conn(0x7f89a40b35e0 msgr2=0x7f89a413b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:09.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.295+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 shutdown_connections 2024-03-29T10:43:09.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.295+0000 7f89aa40c640 1 -- 172.21.15.94:0/2079590384 wait complete. 2024-03-29T10:43:09.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:09.295+0000 7f89aa40c640 1 librados: shutdown 2024-03-29T10:43:09.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:16.102 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:16.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.284+0000 7fc84dda0640 10 monclient: get_monmap_and_config 2024-03-29T10:43:16.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.284+0000 7fc84dda0640 10 monclient: build_initial_monmap 2024-03-29T10:43:16.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.284+0000 7fc84dda0640 10 monclient: monmap: 2024-03-29T10:43:16.267 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:16.267 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:16.285254+0000 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:16.285254+0000 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:16.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.285+0000 7fc84dda0640 1 Processor -- start 2024-03-29T10:43:16.269 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.285+0000 7fc84dda0640 1 -- start start 2024-03-29T10:43:16.269 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.285+0000 7fc84dda0640 10 monclient: init 2024-03-29T10:43:16.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:16.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 10 monclient(hunting): picked mon.noname-a con 0x7fc8480b2800 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 10 monclient(hunting): picked mon.noname-c con 0x7fc848142b20 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 10 monclient(hunting): picked mon.noname-b con 0x7fc84813c040 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8480aad10 con 0x7fc8480b2800 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc848141f30 con 0x7fc848142b20 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8480aaf60 con 0x7fc84813c040 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.286+0000 7fc84dda0640 10 monclient(hunting): authenticate will time out at 1047.554804s 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc84cd9e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc8480b2800 0x7fc848142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:40808/0 (socket says 172.21.15.94:40808) 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc84cd9e640 1 -- 172.21.15.94:0/460321469 learned_addr learned my addr 172.21.15.94:0/460321469 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149566334 0 0) 0x7fc848141f30 con 0x7fc848142b20 2024-03-29T10:43:16.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc8477fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc8477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc8477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc8477fe640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.287+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc838003520 con 0x7fc848142b20 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2815455298 0 0) 0x7fc838003520 con 0x7fc848142b20 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 >> v1:172.21.15.94:6790/0 conn(0x7fc84813c040 legacy=0x7fc84814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 >> v1:172.21.15.94:6789/0 conn(0x7fc8480b2800 legacy=0x7fc848142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 1 monclient: found mon.noname-c 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8480b3900 con 0x7fc848142b20 2024-03-29T10:43:16.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc834001690 con 0x7fc848142b20 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: dump: 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:16.276 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc84dda0640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc84dda0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc834001860 con 0x7fc848142b20 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc8477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc84dda0640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.288+0000 7fc84dda0640 4 set_mon_vals no callback set 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.289+0000 7fc8477fe640 1 -- 172.21.15.94:0/460321469 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc834001cb0 con 0x7fc848142b20 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.289+0000 7fc8477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.289+0000 7fc8477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.289+0000 7fc8477fe640 10 monclient: dump: 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:16.277 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.289+0000 7fc84dda0640 10 monclient: shutdown 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.289+0000 7fc84dda0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.289+0000 7fc84dda0640 1 -- 172.21.15.94:0/460321469 >> v1:172.21.15.173:6789/0 conn(0x7fc848142b20 legacy=0x7fc84813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.290+0000 7fc84dda0640 1 -- 172.21.15.94:0/460321469 shutdown_connections 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.290+0000 7fc84dda0640 1 -- 172.21.15.94:0/460321469 >> 172.21.15.94:0/460321469 conn(0x7fc84806d0e0 msgr2=0x7fc848066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:16.278 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.290+0000 7fc84dda0640 1 -- 172.21.15.94:0/460321469 shutdown_connections 2024-03-29T10:43:16.279 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.290+0000 7fc84dda0640 1 -- 172.21.15.94:0/460321469 wait complete. 2024-03-29T10:43:16.283 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.291+0000 7fc84dda0640 10 monclient: build_initial_monmap 2024-03-29T10:43:16.283 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.291+0000 7fc84dda0640 10 monclient: monmap: 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.291+0000 7fc84dda0640 1 librados: starting msgr at 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.291+0000 7fc84dda0640 1 librados: starting objecter 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.291+0000 7fc84dda0640 1 Processor -- start 2024-03-29T10:43:16.284 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.292+0000 7fc84dda0640 1 -- start start 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.292+0000 7fc84dda0640 1 librados: setting wanted keys 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.292+0000 7fc84dda0640 1 librados: calling monclient init 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.292+0000 7fc84dda0640 10 monclient: init 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 10 monclient(hunting): picked mon.noname-b con 0x7fc8480b2800 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 10 monclient(hunting): picked mon.noname-a con 0x7fc84813c040 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 10 monclient(hunting): picked mon.noname-c con 0x7fc848142b20 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc848145920 con 0x7fc84813c040 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8481dc170 con 0x7fc8480b2800 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc847fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc84813c040 0x7fc8481db350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41338/0 (socket says 172.21.15.94:41338) 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc847fff640 1 -- 172.21.15.94:0/1496378017 learned_addr learned my addr 172.21.15.94:0/1496378017 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:16.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84806e8d0 con 0x7fc848142b20 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc84dda0640 10 monclient(hunting): authenticate will time out at 1047.561483s 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc8457fa640 10 client.?.objecter ms_handle_connect 0x7fc8480b2800 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc8457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc8457fa640 10 client.?.objecter ms_handle_connect 0x7fc848142b20 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc8457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc8457fa640 10 client.?.objecter ms_handle_connect 0x7fc84813c040 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.293+0000 7fc8457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610215155 0 0) 0x7fc8481dc170 con 0x7fc8480b2800 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:43:16.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c004eb0 con 0x7fc8480b2800 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4192367911 0 0) 0x7fc82c004eb0 con 0x7fc8480b2800 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8481dc170 con 0x7fc8480b2800 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc83c001930 con 0x7fc8480b2800 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): dump: 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:16.287 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621753507 0 0) 0x7fc84806e8d0 con 0x7fc848142b20 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.294+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c004eb0 con 0x7fc848142b20 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3975562849 0 0) 0x7fc8481dc170 con 0x7fc8480b2800 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 >> v1:172.21.15.94:6790/0 conn(0x7fc848142b20 legacy=0x7fc8481dba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 >> v1:172.21.15.94:6789/0 conn(0x7fc84813c040 legacy=0x7fc8481db350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:16.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 1 monclient: found mon.b 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848145db0 con 0x7fc8480b2800 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc8457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 10 monclient: _renew_subs 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc848145440 con 0x7fc8480b2800 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 10 monclient: _renew_subs 2024-03-29T10:43:16.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.295+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc848137680 con 0x7fc8480b2800 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc84dda0640 1 librados: init done 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc83c0047e0 con 0x7fc8480b2800 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc8457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc83c004bb0 con 0x7fc8480b2800 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc8457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc8457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc8457fa640 10 monclient: dump: 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:16.290 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.296+0000 7fc846ffd640 4 set_mon_vals no callback set 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.297+0000 7fc84dda0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.297+0000 7fc84dda0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.297+0000 7fc84dda0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.297+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84804f730 con 0x7fc8480b2800 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.298+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fc83c01ab80 con 0x7fc8480b2800 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.299+0000 7fc847fff640 10 monclient: get_auth_request con 0x7fc82c06db20 auth_method 0 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.299+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fc83c084a30 con 0x7fc8480b2800 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.299+0000 7fc8457fa640 10 client.4372.objecter ms_dispatch 0x7fc848000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.299+0000 7fc8457fa640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:16.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.299+0000 7fc8457fa640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:16.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.299+0000 7fc8457fa640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-29T10:43:16.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.299+0000 7fc8457fa640 10 client.4372.objecter ms_handle_connect 0x7fc82c06db20 2024-03-29T10:43:16.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.305+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc83c050310 con 0x7fc8480b2800 2024-03-29T10:43:16.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.305+0000 7fc8457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:16.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.305+0000 7fc8457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:16.714 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.731+0000 7fc84dda0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:16.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.731+0000 7fc84dda0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:16.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.731+0000 7fc84dda0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:16.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.731+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc848145f40 con 0x7fc8480b2800 2024-03-29T10:43:16.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.732+0000 7fc8457fa640 1 -- 172.21.15.94:0/1496378017 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fc848145f40 con 0x7fc8480b2800 2024-03-29T10:43:16.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.732+0000 7fc8457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:16.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.732+0000 7fc8457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:16.716 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:16.716 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:16.720 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 10 librados: watch_flush enter 2024-03-29T10:43:16.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 10 librados: watch_flush exit 2024-03-29T10:43:16.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-29T10:43:16.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 10 client.4372.objecter successfully canceled tick 2024-03-29T10:43:16.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 >> v1:172.21.15.94:6816/3829934808 conn(0x7fc82c06db20 legacy=0x7fc82c06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:16.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 10 monclient: shutdown 2024-03-29T10:43:16.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:16.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 >> v1:172.21.15.173:6789/0 conn(0x7fc8480b2800 legacy=0x7fc848142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:16.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 shutdown_connections 2024-03-29T10:43:16.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.736+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 >> 172.21.15.94:0/1496378017 conn(0x7fc84806d0e0 msgr2=0x7fc8480b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:16.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.737+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 shutdown_connections 2024-03-29T10:43:16.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.737+0000 7fc84dda0640 1 -- 172.21.15.94:0/1496378017 wait complete. 2024-03-29T10:43:16.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:16.737+0000 7fc84dda0640 1 librados: shutdown 2024-03-29T10:43:16.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:23.543 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.722+0000 7f0d262e2640 10 monclient: get_monmap_and_config 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.722+0000 7f0d262e2640 10 monclient: build_initial_monmap 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.722+0000 7f0d262e2640 10 monclient: monmap: 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:23.723764+0000 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:23.723764+0000 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:23.706 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:23.707 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:23.707 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:23.707 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:23.707 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.724+0000 7f0d262e2640 1 Processor -- start 2024-03-29T10:43:23.707 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.724+0000 7f0d262e2640 1 -- start start 2024-03-29T10:43:23.707 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.724+0000 7f0d262e2640 10 monclient: init 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 10 monclient(hunting): picked mon.noname-c con 0x7f0d200b0770 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 10 monclient(hunting): picked mon.noname-b con 0x7f0d2014a700 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 10 monclient(hunting): picked mon.noname-a con 0x7f0d200ac5a0 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d200aba70 con 0x7f0d200ac5a0 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d200aecb0 con 0x7f0d200b0770 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d200af150 con 0x7f0d2014a700 2024-03-29T10:43:23.708 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:23.709 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d262e2640 10 monclient(hunting): authenticate will time out at 1054.993599s 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d1ffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0d200b0770 0x7f0d20149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:48184/0 (socket says 172.21.15.94:48184) 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.725+0000 7f0d1ffff640 1 -- 172.21.15.94:0/2517750809 learned_addr learned my addr 172.21.15.94:0/2517750809 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481971001 0 0) 0x7f0d200aba70 con 0x7f0d200ac5a0 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d08003520 con 0x7f0d200ac5a0 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417442960 0 0) 0x7f0d200aecb0 con 0x7f0d200b0770 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:23.710 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d200aba70 con 0x7f0d200b0770 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1234685563 0 0) 0x7f0d200aba70 con 0x7f0d200b0770 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 >> v1:172.21.15.94:6790/0 conn(0x7f0d2014a700 legacy=0x7f0d200abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.726+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 >> v1:172.21.15.94:6789/0 conn(0x7f0d200ac5a0 legacy=0x7f0d20151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 1 monclient: found mon.noname-c 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d200ac9d0 con 0x7f0d200b0770 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: _finish_auth 0 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:23.711 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d262e2640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0d14001630 con 0x7f0d200b0770 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: dump: 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:23.712 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:23.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d262e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:23.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 1 -- 172.21.15.94:0/2517750809 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d14001800 con 0x7f0d200b0770 2024-03-29T10:43:23.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:23.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d262e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:23.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.727+0000 7f0d262e2640 4 set_mon_vals no callback set 2024-03-29T10:43:23.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.728+0000 7f0d262e2640 10 monclient: shutdown 2024-03-29T10:43:23.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.728+0000 7f0d262e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:23.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.728+0000 7f0d262e2640 1 -- 172.21.15.94:0/2517750809 >> v1:172.21.15.173:6789/0 conn(0x7f0d200b0770 legacy=0x7f0d20149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:23.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.728+0000 7f0d262e2640 1 -- 172.21.15.94:0/2517750809 shutdown_connections 2024-03-29T10:43:23.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.728+0000 7f0d262e2640 1 -- 172.21.15.94:0/2517750809 >> 172.21.15.94:0/2517750809 conn(0x7f0d2006b660 msgr2=0x7f0d2006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:23.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.728+0000 7f0d262e2640 1 -- 172.21.15.94:0/2517750809 shutdown_connections 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.729+0000 7f0d262e2640 1 -- 172.21.15.94:0/2517750809 wait complete. 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.730+0000 7f0d262e2640 10 monclient: build_initial_monmap 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.730+0000 7f0d262e2640 10 monclient: monmap: 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:23.716 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.730+0000 7f0d262e2640 1 librados: starting msgr at 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.730+0000 7f0d262e2640 1 librados: starting objecter 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.730+0000 7f0d262e2640 1 Processor -- start 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.731+0000 7f0d262e2640 1 -- start start 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.731+0000 7f0d262e2640 1 librados: setting wanted keys 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.731+0000 7f0d262e2640 1 librados: calling monclient init 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.731+0000 7f0d262e2640 10 monclient: init 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.731+0000 7f0d262e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.731+0000 7f0d262e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 10 monclient(hunting): picked mon.noname-c con 0x7f0d200ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f0d200ac5a0 0x7f0d20126150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35182/0 (socket says 172.21.15.94:35182) 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1ffff640 1 -- 172.21.15.94:0/2454025154 learned_addr learned my addr 172.21.15.94:0/2454025154 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 10 monclient(hunting): picked mon.noname-b con 0x7f0d200b0770 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 10 monclient(hunting): picked mon.noname-a con 0x7f0d2014a700 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:23.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d201239d0 con 0x7f0d2014a700 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1cff9640 10 client.?.objecter ms_handle_connect 0x7f0d200ac5a0 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d20126860 con 0x7f0d200b0770 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d20128540 con 0x7f0d200ac5a0 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d262e2640 10 monclient(hunting): authenticate will time out at 1055.000515s 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1cff9640 10 client.?.objecter ms_handle_connect 0x7f0d2014a700 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1cff9640 10 client.?.objecter ms_handle_connect 0x7f0d200b0770 2024-03-29T10:43:23.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.732+0000 7f0d1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:23.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.734+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282055658 0 0) 0x7f0d201239d0 con 0x7f0d2014a700 2024-03-29T10:43:23.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.734+0000 7f0d1cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:23.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.734+0000 7f0d1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:23.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.734+0000 7f0d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:23.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.734+0000 7f0d1cff9640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:43:23.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c004eb0 con 0x7f0d2014a700 2024-03-29T10:43:23.727 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595638087 0 0) 0x7f0d20128540 con 0x7f0d200ac5a0 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d201239d0 con 0x7f0d200ac5a0 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915208973 0 0) 0x7f0d20126860 con 0x7f0d200b0770 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.735+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d20128540 con 0x7f0d200b0770 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3605268491 0 0) 0x7f0d0c004eb0 con 0x7f0d2014a700 2024-03-29T10:43:23.728 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d20126860 con 0x7f0d2014a700 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2977885168 0 0) 0x7f0d201239d0 con 0x7f0d200ac5a0 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d0c004eb0 con 0x7f0d200ac5a0 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0d18001c00 con 0x7f0d2014a700 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:23.729 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0d14004980 con 0x7f0d200ac5a0 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:23.730 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031814297 0 0) 0x7f0d20128540 con 0x7f0d200b0770 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d201239d0 con 0x7f0d200b0770 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0d100017c0 con 0x7f0d200b0770 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.731 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1080905826 0 0) 0x7f0d20126860 con 0x7f0d2014a700 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.736+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 >> v1:172.21.15.94:6790/0 conn(0x7f0d200ac5a0 legacy=0x7f0d20126150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 >> v1:172.21.15.173:6789/0 conn(0x7f0d200b0770 legacy=0x7f0d201dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 1 monclient: found mon.a 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d20126ed0 con 0x7f0d2014a700 2024-03-29T10:43:23.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 5 monclient: authenticate success, global_id 4395 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 10 monclient: _renew_subs 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d20127120 con 0x7f0d2014a700 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 10 monclient: _renew_subs 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d201dc3b0 con 0x7f0d2014a700 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d262e2640 1 librados: init done 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d18003990 con 0x7f0d2014a700 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:23.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0d18003ca0 con 0x7f0d2014a700 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1e7fc640 4 set_mon_vals no callback set 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.737+0000 7f0d1cff9640 10 monclient: dump: 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:23.734 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.739+0000 7f0d262e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.739+0000 7f0d262e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.739+0000 7f0d262e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.739+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f0d18019c70 con 0x7f0d2014a700 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.739+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d200ab560 con 0x7f0d2014a700 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.740+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0d18083b70 con 0x7f0d2014a700 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.740+0000 7f0d1cff9640 10 client.4395.objecter ms_dispatch 0x7f0d20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.740+0000 7f0d1cff9640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:23.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.740+0000 7f0d1cff9640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:23.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.740+0000 7f0d1cff9640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T10:43:23.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.741+0000 7f0d1ffff640 10 monclient: get_auth_request con 0x7f0d0c06e100 auth_method 0 2024-03-29T10:43:23.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.741+0000 7f0d1cff9640 10 client.4395.objecter ms_handle_connect 0x7f0d0c06e100 2024-03-29T10:43:23.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.750+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d18083e60 con 0x7f0d2014a700 2024-03-29T10:43:23.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.750+0000 7f0d1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:23.737 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:23.750+0000 7f0d1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:24.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.165+0000 7f0d262e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:24.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.165+0000 7f0d262e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:24.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.165+0000 7f0d262e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:43:24.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.165+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d20128380 con 0x7f0d2014a700 2024-03-29T10:43:24.149 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.166+0000 7f0d1cff9640 1 -- 172.21.15.94:0/2454025154 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f0d1804f450 con 0x7f0d2014a700 2024-03-29T10:43:24.149 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.166+0000 7f0d1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:24.149 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.166+0000 7f0d1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:24.149 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:24.149 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:24.153 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 10 librados: watch_flush enter 2024-03-29T10:43:24.153 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 10 librados: watch_flush exit 2024-03-29T10:43:24.153 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T10:43:24.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 10 client.4395.objecter successfully canceled tick 2024-03-29T10:43:24.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 >> v1:172.21.15.94:6816/3829934808 conn(0x7f0d0c06e100 legacy=0x7f0d0c0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:24.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 10 monclient: shutdown 2024-03-29T10:43:24.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:24.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.170+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 >> v1:172.21.15.94:6789/0 conn(0x7f0d2014a700 legacy=0x7f0d201e6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:24.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.171+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 shutdown_connections 2024-03-29T10:43:24.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.171+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 >> 172.21.15.94:0/2454025154 conn(0x7f0d2006b660 msgr2=0x7f0d200b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:24.155 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.171+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 shutdown_connections 2024-03-29T10:43:24.155 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.171+0000 7f0d262e2640 1 -- 172.21.15.94:0/2454025154 wait complete. 2024-03-29T10:43:24.155 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:24.171+0000 7f0d262e2640 1 librados: shutdown 2024-03-29T10:43:24.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:30.974 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:31.110 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.127+0000 7f6457c5c640 10 monclient: get_monmap_and_config 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.127+0000 7f6457c5c640 10 monclient: build_initial_monmap 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.127+0000 7f6457c5c640 10 monclient: monmap: 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:31.128416+0000 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:31.128416+0000 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:31.111 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.128+0000 7f6457c5c640 1 Processor -- start 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.128+0000 7f6457c5c640 1 -- start start 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.128+0000 7f6457c5c640 10 monclient: init 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 10 monclient(hunting): picked mon.noname-b con 0x7f6450071de0 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 10 monclient(hunting): picked mon.noname-a con 0x7f64500721c0 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 10 monclient(hunting): picked mon.noname-c con 0x7f6450145ae0 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6450145650 con 0x7f64500721c0 2024-03-29T10:43:31.114 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f64559f8640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f6450071de0 0x7f64500aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:43154/0 (socket says 172.21.15.94:43154) 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f64559f8640 1 -- 172.21.15.94:0/345877476 learned_addr learned my addr 172.21.15.94:0/345877476 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 1 -- 172.21.15.94:0/345877476 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64500ae260 con 0x7f6450145ae0 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 1 -- 172.21.15.94:0/345877476 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64501420d0 con 0x7f6450071de0 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.129+0000 7f6457c5c640 10 monclient(hunting): authenticate will time out at 1062.398056s 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225792688 0 0) 0x7f64500ae260 con 0x7f6450145ae0 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): _negotiate 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643c003520 con 0x7f6450145ae0 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201233727 0 0) 0x7f64501420d0 con 0x7f6450071de0 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): _negotiate 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:31.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:31.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:43:31.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643c003840 con 0x7f6450071de0 2024-03-29T10:43:31.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.130+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1825586413 0 0) 0x7f643c003840 con 0x7f6450071de0 2024-03-29T10:43:31.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.131+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 >> v1:172.21.15.173:6789/0 conn(0x7f6450145ae0 legacy=0x7f6450145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:31.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.131+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 >> v1:172.21.15.94:6789/0 conn(0x7f64500721c0 legacy=0x7f645013f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 10 monclient: _finish_hunting 0 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 1 monclient: found mon.noname-b 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64500b0590 con 0x7f6450071de0 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 10 monclient: _finish_auth 0 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 10 monclient: _check_auth_tickets 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:31.117 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f644c001630 con 0x7f6450071de0 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f64549f6640 10 monclient: dump: 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:31.118 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f6457c5c640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.132+0000 7f6457c5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f644c001810 con 0x7f6450071de0 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f64549f6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f6457c5c640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f6457c5c640 4 set_mon_vals no callback set 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f64549f6640 1 -- 172.21.15.94:0/345877476 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f644c001ba0 con 0x7f6450071de0 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f64549f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f64549f6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.133+0000 7f64549f6640 10 monclient: dump: 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.119 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.134+0000 7f6457c5c640 10 monclient: shutdown 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.134+0000 7f6457c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.134+0000 7f6457c5c640 1 -- 172.21.15.94:0/345877476 >> v1:172.21.15.94:6790/0 conn(0x7f6450071de0 legacy=0x7f64500aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.134+0000 7f6457c5c640 1 -- 172.21.15.94:0/345877476 shutdown_connections 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.134+0000 7f6457c5c640 1 -- 172.21.15.94:0/345877476 >> 172.21.15.94:0/345877476 conn(0x7f645006b660 msgr2=0x7f645006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.134+0000 7f6457c5c640 1 -- 172.21.15.94:0/345877476 shutdown_connections 2024-03-29T10:43:31.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.134+0000 7f6457c5c640 1 -- 172.21.15.94:0/345877476 wait complete. 2024-03-29T10:43:31.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 10 monclient: build_initial_monmap 2024-03-29T10:43:31.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 10 monclient: monmap: 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 1 librados: starting msgr at 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 1 librados: starting objecter 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 1 Processor -- start 2024-03-29T10:43:31.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 1 -- start start 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 1 librados: setting wanted keys 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 1 librados: calling monclient init 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.135+0000 7f6457c5c640 10 monclient: init 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.136+0000 7f6457c5c640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.136+0000 7f6457c5c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f6457c5c640 10 monclient(hunting): picked mon.noname-a con 0x7f6450071de0 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f6457c5c640 10 monclient(hunting): picked mon.noname-b con 0x7f6450145ae0 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f6457c5c640 10 monclient(hunting): picked mon.noname-c con 0x7f6450132e90 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f64561f9640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f6450132e90 0x7f64501da720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:44898/0 (socket says 172.21.15.94:44898) 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f64561f9640 1 -- 172.21.15.94:0/657300492 learned_addr learned my addr 172.21.15.94:0/657300492 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6450135740 con 0x7f6450071de0 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6450135a20 con 0x7f6450145ae0 2024-03-29T10:43:31.126 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f64467fc640 10 client.?.objecter ms_handle_connect 0x7f6450132e90 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f64467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6450134420 con 0x7f6450132e90 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f64467fc640 10 client.?.objecter ms_handle_connect 0x7f6450145ae0 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.137+0000 7f64467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.138+0000 7f6457c5c640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.138+0000 7f6457c5c640 10 monclient(hunting): authenticate will time out at 1062.406154s 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934699412 0 0) 0x7f6450135a20 con 0x7f6450145ae0 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6434004eb0 con 0x7f6450145ae0 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524796255 0 0) 0x7f6450134420 con 0x7f6450132e90 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:31.127 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6450135a20 con 0x7f6450132e90 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2717058339 0 0) 0x7f6434004eb0 con 0x7f6450145ae0 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6450134420 con 0x7f6450145ae0 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6440001870 con 0x7f6450145ae0 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): dump: 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:31.128 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527214099 0 0) 0x7f6450135a20 con 0x7f6450132e90 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6434004eb0 con 0x7f6450132e90 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6448001c70 con 0x7f6450132e90 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.139+0000 7f64467fc640 10 monclient(hunting): dump: 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:31.129 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.140+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1891049510 0 0) 0x7f6434004eb0 con 0x7f6450132e90 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.140+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 >> v1:172.21.15.173:6789/0 conn(0x7f6450145ae0 legacy=0x7f6450132780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.140+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 >> v1:172.21.15.94:6789/0 conn(0x7f6450071de0 legacy=0x7f6450135e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.140+0000 7f64467fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.140+0000 7f64467fc640 1 monclient: found mon.c 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6450133d90 con 0x7f6450132e90 2024-03-29T10:43:31.130 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 10 monclient: _renew_subs 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6450134040 con 0x7f6450132e90 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6448003970 con 0x7f6450132e90 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6448003d40 con 0x7f6450132e90 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f64467fc640 10 monclient: dump: 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:31.131 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.132 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:31.132 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:31.132 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:31.132 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:31.132 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:31.132 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:31.132 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:31.133 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6447fff640 4 set_mon_vals no callback set 2024-03-29T10:43:31.133 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:31.133 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 10 monclient: _renew_subs 2024-03-29T10:43:31.133 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:31.133 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64501db010 con 0x7f6450132e90 2024-03-29T10:43:31.133 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.141+0000 7f6457c5c640 1 librados: init done 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.142+0000 7f6457c5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.142+0000 7f6457c5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.142+0000 7f6457c5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.143+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6450134280 con 0x7f6450132e90 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.144+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f6448003ea0 con 0x7f6450132e90 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.144+0000 7f64559f8640 10 monclient: get_auth_request con 0x7f643406de40 auth_method 0 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.144+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f6448084c80 con 0x7f6450132e90 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.144+0000 7f64467fc640 10 client.4397.objecter ms_dispatch 0x7f6450000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.144+0000 7f64467fc640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:31.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.144+0000 7f64467fc640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:31.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.144+0000 7f64467fc640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:43:31.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.145+0000 7f64467fc640 10 client.4397.objecter ms_handle_connect 0x7f643406de40 2024-03-29T10:43:31.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.152+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6450134280 con 0x7f6450132e90 2024-03-29T10:43:31.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.152+0000 7f64467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:31.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.152+0000 7f64467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:31.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.475+0000 7f6457c5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:31.458 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.475+0000 7f6457c5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:31.459 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.475+0000 7f6457c5c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:31.459 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.475+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6450134280 con 0x7f6450132e90 2024-03-29T10:43:31.461 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:31.461 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:31.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.476+0000 7f64467fc640 1 -- 172.21.15.94:0/657300492 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f6450134280 con 0x7f6450132e90 2024-03-29T10:43:31.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.476+0000 7f64467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:31.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.476+0000 7f64467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:31.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.479+0000 7f6457c5c640 10 librados: watch_flush enter 2024-03-29T10:43:31.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.480+0000 7f6457c5c640 10 librados: watch_flush exit 2024-03-29T10:43:31.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.480+0000 7f6457c5c640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:43:31.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.480+0000 7f6457c5c640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:43:31.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 >> v1:172.21.15.94:6816/3829934808 conn(0x7f643406de40 legacy=0x7f6434070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 10 monclient: shutdown 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 >> v1:172.21.15.94:6790/0 conn(0x7f6450132e90 legacy=0x7f64501da720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 shutdown_connections 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 >> 172.21.15.94:0/657300492 conn(0x7f645006b660 msgr2=0x7f64500b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 shutdown_connections 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 1 -- 172.21.15.94:0/657300492 wait complete. 2024-03-29T10:43:31.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:31.482+0000 7f6457c5c640 1 librados: shutdown 2024-03-29T10:43:31.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:38.289 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.456+0000 7f066cde2640 10 monclient: get_monmap_and_config 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.456+0000 7f066cde2640 10 monclient: build_initial_monmap 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.457+0000 7f066cde2640 10 monclient: monmap: 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:38.458880+0000 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:38.458880+0000 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:38.441 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:38.442 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:38.442 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:38.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.457+0000 7f066cde2640 1 Processor -- start 2024-03-29T10:43:38.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.457+0000 7f066cde2640 1 -- start start 2024-03-29T10:43:38.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.457+0000 7f066cde2640 10 monclient: init 2024-03-29T10:43:38.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 10 monclient(hunting): picked mon.noname-b con 0x7f06680b1f60 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 10 monclient(hunting): picked mon.noname-c con 0x7f0668071eb0 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 10 monclient(hunting): picked mon.noname-a con 0x7f0668143d10 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066806ef80 con 0x7f0668143d10 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066806f260 con 0x7f0668071eb0 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06680b40c0 con 0x7f06680b1f60 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:38.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.458+0000 7f066cde2640 10 monclient(hunting): authenticate will time out at 1069.728077s 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.461+0000 7f066659c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f06680b1f60 0x7f0668143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:50074/0 (socket says 172.21.15.94:50074) 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.461+0000 7f066659c640 1 -- 172.21.15.94:0/1950049098 learned_addr learned my addr 172.21.15.94:0/1950049098 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.461+0000 7f0666d9d640 1 --1- 172.21.15.94:0/1950049098 >> v1:172.21.15.94:6789/0 conn(0x7f0668143d10 0x7f0668146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54963/0 (socket says 172.21.15.94:54963) 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958778504 0 0) 0x7f06680b40c0 con 0x7f06680b1f60 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0650003520 con 0x7f06680b1f60 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959326639 0 0) 0x7f066806ef80 con 0x7f0668143d10 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): _negotiate 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:38.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06680b40c0 con 0x7f0668143d10 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616035815 0 0) 0x7f0650003520 con 0x7f06680b1f60 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 >> v1:172.21.15.173:6789/0 conn(0x7f0668071eb0 legacy=0x7f0668146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 >> v1:172.21.15.94:6789/0 conn(0x7f0668143d10 legacy=0x7f0668146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient: _finish_hunting 0 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 monclient: found mon.noname-b 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0668155d70 con 0x7f06680b1f60 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient: _finish_auth 0 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient: _check_auth_tickets 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:38.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0658001690 con 0x7f06680b1f60 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066559a640 10 monclient: dump: 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066cde2640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.462+0000 7f066cde2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:38.453 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06580017f0 con 0x7f06680b1f60 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066559a640 1 -- 172.21.15.94:0/1950049098 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0658001c40 con 0x7f06680b1f60 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066559a640 10 monclient: dump: 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:38.454 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066cde2640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.463+0000 7f066cde2640 4 set_mon_vals no callback set 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.464+0000 7f066cde2640 10 monclient: shutdown 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.464+0000 7f066cde2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.464+0000 7f066cde2640 1 -- 172.21.15.94:0/1950049098 >> v1:172.21.15.94:6790/0 conn(0x7f06680b1f60 legacy=0x7f0668143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.465+0000 7f066cde2640 1 -- 172.21.15.94:0/1950049098 shutdown_connections 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.465+0000 7f066cde2640 1 -- 172.21.15.94:0/1950049098 >> 172.21.15.94:0/1950049098 conn(0x7f0668066a00 msgr2=0x7f0668066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.465+0000 7f066cde2640 1 -- 172.21.15.94:0/1950049098 shutdown_connections 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.465+0000 7f066cde2640 1 -- 172.21.15.94:0/1950049098 wait complete. 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.466+0000 7f066cde2640 10 monclient: build_initial_monmap 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.466+0000 7f066cde2640 10 monclient: monmap: 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:38.455 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 1 librados: starting msgr at 2024-03-29T10:43:38.456 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 1 librados: starting objecter 2024-03-29T10:43:38.459 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 1 Processor -- start 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 1 -- start start 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 1 librados: setting wanted keys 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 1 librados: calling monclient init 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 10 monclient: init 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.467+0000 7f066cde2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 10 monclient(hunting): picked mon.noname-c con 0x7f06680b1f60 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066659c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f06680b1f60 0x7f06681decc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:50772/0 (socket says 172.21.15.94:50772) 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 10 monclient(hunting): picked mon.noname-b con 0x7f0668143d10 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 10 monclient(hunting): picked mon.noname-a con 0x7f066814a050 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06681498e0 con 0x7f066814a050 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066814aec0 con 0x7f0668143d10 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066813bc00 con 0x7f06680b1f60 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:38.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066cde2640 10 monclient(hunting): authenticate will time out at 1069.737302s 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f066659c640 1 -- 172.21.15.94:0/1457637566 learned_addr learned my addr 172.21.15.94:0/1457637566 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 10 client.?.objecter ms_handle_connect 0x7f06680b1f60 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487418529 0 0) 0x7f066813bc00 con 0x7f06680b1f60 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.468+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0648004eb0 con 0x7f06680b1f60 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.470+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3062032908 0 0) 0x7f0648004eb0 con 0x7f06680b1f60 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f066813bc00 con 0x7f06680b1f60 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0658001b00 con 0x7f06680b1f60 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:38.461 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 client.?.objecter ms_handle_connect 0x7f0668143d10 2024-03-29T10:43:38.462 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818512266 0 0) 0x7f066814aec0 con 0x7f0668143d10 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0648004eb0 con 0x7f0668143d10 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2627614295 0 0) 0x7f066813bc00 con 0x7f06680b1f60 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 >> v1:172.21.15.173:6789/0 conn(0x7f0668143d10 legacy=0x7f066813ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.471+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 >> v1:172.21.15.94:6789/0 conn(0x7f066814a050 legacy=0x7f066813b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 1 monclient: found mon.c 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066814b130 con 0x7f06680b1f60 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:38.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 10 monclient: _renew_subs 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0668149b30 con 0x7f06680b1f60 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 10 monclient: _renew_subs 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06680708e0 con 0x7f06680b1f60 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f066cde2640 1 librados: init done 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0658004bf0 con 0x7f06680b1f60 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0656ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.472+0000 7f0664d99640 4 set_mon_vals no callback set 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.473+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0658001c60 con 0x7f06680b1f60 2024-03-29T10:43:38.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.473+0000 7f0656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.473+0000 7f0656ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.473+0000 7f0656ffd640 10 monclient: dump: 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.475+0000 7f066cde2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.475+0000 7f066cde2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:38.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.475+0000 7f066cde2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:38.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.475+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c005080 con 0x7f06680b1f60 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.478+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f0658004f00 con 0x7f06680b1f60 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.478+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0658084c20 con 0x7f06680b1f60 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.478+0000 7f0656ffd640 10 client.4403.objecter ms_dispatch 0x7f0668000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.478+0000 7f0656ffd640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.478+0000 7f0656ffd640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.478+0000 7f0656ffd640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.478+0000 7f0665d9b640 10 monclient: get_auth_request con 0x7f064806dc40 auth_method 0 2024-03-29T10:43:38.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.479+0000 7f0656ffd640 10 client.4403.objecter ms_handle_connect 0x7f064806dc40 2024-03-29T10:43:38.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.484+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0658050500 con 0x7f06680b1f60 2024-03-29T10:43:38.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.484+0000 7f0656ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:38.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.484+0000 7f0656ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:38.791 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:38.791 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:38.791 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.805+0000 7f066cde2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:38.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.805+0000 7f066cde2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:38.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.805+0000 7f066cde2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:38.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.805+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f064c005330 con 0x7f06680b1f60 2024-03-29T10:43:38.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.806+0000 7f0656ffd640 1 -- 172.21.15.94:0/1457637566 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f0658052430 con 0x7f06680b1f60 2024-03-29T10:43:38.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.806+0000 7f0656ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:38.792 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.806+0000 7f0656ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:38.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.809+0000 7f066cde2640 10 librados: watch_flush enter 2024-03-29T10:43:38.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.809+0000 7f066cde2640 10 librados: watch_flush exit 2024-03-29T10:43:38.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.809+0000 7f066cde2640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-29T10:43:38.793 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.809+0000 7f066cde2640 10 client.4403.objecter successfully canceled tick 2024-03-29T10:43:38.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 >> v1:172.21.15.94:6816/3829934808 conn(0x7f064806dc40 legacy=0x7f0648070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:38.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 10 monclient: shutdown 2024-03-29T10:43:38.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:38.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 >> v1:172.21.15.94:6790/0 conn(0x7f06680b1f60 legacy=0x7f06681decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:38.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 shutdown_connections 2024-03-29T10:43:38.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 >> 172.21.15.94:0/1457637566 conn(0x7f0668066a00 msgr2=0x7f0668066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:38.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 shutdown_connections 2024-03-29T10:43:38.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 1 -- 172.21.15.94:0/1457637566 wait complete. 2024-03-29T10:43:38.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:38.810+0000 7f066cde2640 1 librados: shutdown 2024-03-29T10:43:38.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:45.617 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.780+0000 7f6f86b79640 10 monclient: get_monmap_and_config 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.780+0000 7f6f86b79640 10 monclient: build_initial_monmap 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.780+0000 7f6f86b79640 10 monclient: monmap: 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:45.781816+0000 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:45.781816+0000 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:45.764 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:45.765 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:45.765 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:45.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.781+0000 7f6f86b79640 1 Processor -- start 2024-03-29T10:43:45.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.782+0000 7f6f86b79640 1 -- start start 2024-03-29T10:43:45.765 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.782+0000 7f6f86b79640 10 monclient: init 2024-03-29T10:43:45.766 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.783+0000 7f6f86b79640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:45.766 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.783+0000 7f6f86b79640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:45.770 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.783+0000 7f6f86b79640 10 monclient(hunting): picked mon.noname-a con 0x7f6f800a9750 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:45.770 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f86b79640 10 monclient(hunting): picked mon.noname-c con 0x7f6f8013e1e0 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:45.770 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f86b79640 10 monclient(hunting): picked mon.noname-b con 0x7f6f80071c30 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:45.770 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f86b79640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800ab300 con 0x7f6f800a9750 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f86b79640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800ac2f0 con 0x7f6f8013e1e0 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f86b79640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800ad8d0 con 0x7f6f80071c30 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f86b79640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f86b79640 10 monclient(hunting): authenticate will time out at 1077.053335s 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f85116640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f6f80071c30 0x7f6f800ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:39524/0 (socket says 172.21.15.94:39524) 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.785+0000 7f6f85116640 1 -- 172.21.15.94:0/1854675812 learned_addr learned my addr 172.21.15.94:0/1854675812 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472738001 0 0) 0x7f6f800ad8d0 con 0x7f6f80071c30 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70003440 con 0x7f6f80071c30 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1004297904 0 0) 0x7f6f70003440 con 0x7f6f80071c30 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 >> v1:172.21.15.173:6789/0 conn(0x7f6f8013e1e0 legacy=0x7f6f800ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:45.771 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 >> v1:172.21.15.94:6789/0 conn(0x7f6f800a9750 legacy=0x7f6f8013dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f800ada60 con 0x7f6f80071c30 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6f780019f0 con 0x7f6f80071c30 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.786+0000 7f6f7f7fe640 10 monclient: dump: 2024-03-29T10:43:45.772 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.787+0000 7f6f86b79640 5 monclient: authenticate success, global_id 4406 2024-03-29T10:43:45.773 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.787+0000 7f6f86b79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:45.774 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.788+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f78001c50 con 0x7f6f80071c30 2024-03-29T10:43:45.774 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.788+0000 7f6f7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.788+0000 7f6f7f7fe640 1 -- 172.21.15.94:0/1854675812 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6f780020a0 con 0x7f6f80071c30 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.788+0000 7f6f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.788+0000 7f6f7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.788+0000 7f6f7f7fe640 10 monclient: dump: 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:45.775 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.789+0000 7f6f86b79640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.789+0000 7f6f86b79640 4 set_mon_vals no callback set 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.790+0000 7f6f86b79640 10 monclient: shutdown 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.790+0000 7f6f86b79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.790+0000 7f6f86b79640 1 -- 172.21.15.94:0/1854675812 >> v1:172.21.15.94:6790/0 conn(0x7f6f80071c30 legacy=0x7f6f800ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.790+0000 7f6f86b79640 1 -- 172.21.15.94:0/1854675812 shutdown_connections 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.790+0000 7f6f86b79640 1 -- 172.21.15.94:0/1854675812 >> 172.21.15.94:0/1854675812 conn(0x7f6f8006b660 msgr2=0x7f6f8006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.791+0000 7f6f86b79640 1 -- 172.21.15.94:0/1854675812 shutdown_connections 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.791+0000 7f6f86b79640 1 -- 172.21.15.94:0/1854675812 wait complete. 2024-03-29T10:43:45.776 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 10 monclient: build_initial_monmap 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 10 monclient: monmap: 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 1 librados: starting msgr at 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 1 librados: starting objecter 2024-03-29T10:43:45.777 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 1 Processor -- start 2024-03-29T10:43:45.778 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 1 -- start start 2024-03-29T10:43:45.778 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 1 librados: setting wanted keys 2024-03-29T10:43:45.778 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 1 librados: calling monclient init 2024-03-29T10:43:45.778 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.792+0000 7f6f86b79640 10 monclient: init 2024-03-29T10:43:45.778 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.793+0000 7f6f86b79640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:45.778 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.793+0000 7f6f86b79640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:45.778 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.793+0000 7f6f86b79640 10 monclient(hunting): picked mon.noname-b con 0x7f6f80071c30 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f86b79640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a9750 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f86b79640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8013e1e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f86b79640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f80142560 con 0x7f6f8013e1e0 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f86b79640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f80141fa0 con 0x7f6f80071c30 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f86b79640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8008bca0 con 0x7f6f800a9750 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f86b79640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f86b79640 10 monclient(hunting): authenticate will time out at 1077.062986s 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f7ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f6f800a9750 0x7f6f80137d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41632/0 (socket says 172.21.15.94:41632) 2024-03-29T10:43:45.779 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.794+0000 7f6f7ffff640 1 -- 172.21.15.94:0/4160839025 learned_addr learned my addr 172.21.15.94:0/4160839025 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:45.780 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.795+0000 7f6f7d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f800a9750 2024-03-29T10:43:45.780 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.795+0000 7f6f7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:45.781 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145512761 0 0) 0x7f6f8008bca0 con 0x7f6f800a9750 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f64004eb0 con 0x7f6f800a9750 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3633033068 0 0) 0x7f6f64004eb0 con 0x7f6f800a9750 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.796+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f8008bca0 con 0x7f6f800a9750 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.797+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6f6c0017d0 con 0x7f6f800a9750 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.797+0000 7f6f7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.797+0000 7f6f7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.797+0000 7f6f7d7fa640 10 monclient(hunting): dump: 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:45.782 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.797+0000 7f6f7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.797+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3505388364 0 0) 0x7f6f8008bca0 con 0x7f6f800a9750 2024-03-29T10:43:45.783 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.797+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 >> v1:172.21.15.173:6789/0 conn(0x7f6f80071c30 legacy=0x7f6f8013f940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:45.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 >> v1:172.21.15.94:6789/0 conn(0x7f6f8013e1e0 legacy=0x7f6f801eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:45.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:45.784 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 1 monclient: found mon.c 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8008be30 con 0x7f6f800a9750 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f8013e1e0 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f6c001930 con 0x7f6f800a9750 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6f6c004a40 con 0x7f6f800a9750 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:45.785 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7d7fa640 10 monclient: dump: 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:45.786 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.801+0000 7f6f7effd640 4 set_mon_vals no callback set 2024-03-29T10:43:45.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 5 monclient: authenticate success, global_id 4409 2024-03-29T10:43:45.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 10 monclient: _renew_subs 2024-03-29T10:43:45.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:45.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f801363c0 con 0x7f6f800a9750 2024-03-29T10:43:45.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:45.787 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 10 monclient: _renew_subs 2024-03-29T10:43:45.788 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:45.788 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f80140050 con 0x7f6f800a9750 2024-03-29T10:43:45.788 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.802+0000 7f6f86b79640 1 librados: init done 2024-03-29T10:43:45.788 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.804+0000 7f6f86b79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:45.788 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.804+0000 7f6f86b79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:45.788 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.804+0000 7f6f86b79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:45.788 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.804+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f800abb00 con 0x7f6f800a9750 2024-03-29T10:43:45.790 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.807+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f6f6c01aa10 con 0x7f6f800a9750 2024-03-29T10:43:45.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.808+0000 7f6f84915640 10 monclient: get_auth_request con 0x7f6f6406dbd0 auth_method 0 2024-03-29T10:43:45.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.808+0000 7f6f7d7fa640 10 client.4409.objecter ms_handle_connect 0x7f6f6406dbd0 2024-03-29T10:43:45.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.808+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f6f6c084aa0 con 0x7f6f800a9750 2024-03-29T10:43:45.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.808+0000 7f6f7d7fa640 10 client.4409.objecter ms_dispatch 0x7f6f80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:45.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.808+0000 7f6f7d7fa640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:45.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.808+0000 7f6f7d7fa640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:45.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.808+0000 7f6f7d7fa640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T10:43:45.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.815+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f6c0501b0 con 0x7f6f800a9750 2024-03-29T10:43:45.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.815+0000 7f6f7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:45.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:45.815+0000 7f6f7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:46.131 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:46.131 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:46.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.147+0000 7f6f86b79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:46.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.147+0000 7f6f86b79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:46.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.147+0000 7f6f86b79640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:43:46.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.147+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f80141da0 con 0x7f6f800a9750 2024-03-29T10:43:46.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.148+0000 7f6f7d7fa640 1 -- 172.21.15.94:0/4160839025 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f6f6c0522b0 con 0x7f6f800a9750 2024-03-29T10:43:46.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.148+0000 7f6f7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:46.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.148+0000 7f6f7d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:46.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.151+0000 7f6f86b79640 10 librados: watch_flush enter 2024-03-29T10:43:46.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 10 librados: watch_flush exit 2024-03-29T10:43:46.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T10:43:46.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 10 client.4409.objecter successfully canceled tick 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 >> v1:172.21.15.94:6816/3829934808 conn(0x7f6f6406dbd0 legacy=0x7f6f64070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 10 monclient: shutdown 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 >> v1:172.21.15.94:6790/0 conn(0x7f6f800a9750 legacy=0x7f6f80137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 shutdown_connections 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 >> 172.21.15.94:0/4160839025 conn(0x7f6f8006b660 msgr2=0x7f6f800adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 shutdown_connections 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.152+0000 7f6f86b79640 1 -- 172.21.15.94:0/4160839025 wait complete. 2024-03-29T10:43:46.136 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:46.153+0000 7f6f86b79640 1 librados: shutdown 2024-03-29T10:43:46.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:52.960 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.129+0000 7fa501ac9640 10 monclient: get_monmap_and_config 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.129+0000 7fa501ac9640 10 monclient: build_initial_monmap 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.129+0000 7fa501ac9640 10 monclient: monmap: 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:43:53.130126+0000 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:43:53.130126+0000 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:53.112 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:53.113 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:53.113 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:43:53.113 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:43:53.113 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.130+0000 7fa501ac9640 1 Processor -- start 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.130+0000 7fa501ac9640 1 -- start start 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.130+0000 7fa501ac9640 10 monclient: init 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc0b4790 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fa4fc0aea90 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fa4fc0b39f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:53.115 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc0b0c60 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc0af020 con 0x7fa4fc0b4790 2024-03-29T10:43:53.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc13fbd0 con 0x7fa4fc0aea90 2024-03-29T10:43:53.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:53.116 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.132+0000 7fa501ac9640 10 monclient(hunting): authenticate will time out at 1084.400683s 2024-03-29T10:43:53.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.136+0000 7fa4fb7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fa4fc0b4790 0x7fa4fc0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:34369/0 (socket says 172.21.15.94:34369) 2024-03-29T10:43:53.119 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.136+0000 7fa4fb7fe640 1 -- 172.21.15.94:0/3227920090 learned_addr learned my addr 172.21.15.94:0/3227920090 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433028172 0 0) 0x7fa4fc0af020 con 0x7fa4fc0b4790 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e4003520 con 0x7fa4fc0b4790 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874664314 0 0) 0x7fa4fc0b0c60 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:53.120 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:53.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:43:53.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4fc0af020 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1838820922 0 0) 0x7fa4fc0af020 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 >> v1:172.21.15.94:6790/0 conn(0x7fa4fc0aea90 legacy=0x7fa4fc0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:53.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 >> v1:172.21.15.173:6789/0 conn(0x7fa4fc0b4790 legacy=0x7fa4fc0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:53.121 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 monclient: found mon.noname-a 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc13fd60 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa501ac9640 5 monclient: authenticate success, global_id 4404 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa501ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa4f40019f0 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.137+0000 7fa4fa7fc640 10 monclient: dump: 2024-03-29T10:43:53.122 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.138+0000 7fa501ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.138+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f4001c10 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.123 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.138+0000 7fa4fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.138+0000 7fa501ac9640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.138+0000 7fa501ac9640 4 set_mon_vals no callback set 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.138+0000 7fa4fa7fc640 1 -- 172.21.15.94:0/3227920090 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa4f4001fa0 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.138+0000 7fa4fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.139+0000 7fa4fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.139+0000 7fa4fa7fc640 10 monclient: dump: 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:53.124 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:53.125 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:53.125 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:53.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.139+0000 7fa501ac9640 10 monclient: shutdown 2024-03-29T10:43:53.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.139+0000 7fa501ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:53.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.139+0000 7fa501ac9640 1 -- 172.21.15.94:0/3227920090 >> v1:172.21.15.94:6789/0 conn(0x7fa4fc0b39f0 legacy=0x7fa4fc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:53.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.139+0000 7fa501ac9640 1 -- 172.21.15.94:0/3227920090 shutdown_connections 2024-03-29T10:43:53.125 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.139+0000 7fa501ac9640 1 -- 172.21.15.94:0/3227920090 >> 172.21.15.94:0/3227920090 conn(0x7fa4fc066a00 msgr2=0x7fa4fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:53.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.142+0000 7fa501ac9640 1 -- 172.21.15.94:0/3227920090 shutdown_connections 2024-03-29T10:43:53.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.142+0000 7fa501ac9640 1 -- 172.21.15.94:0/3227920090 wait complete. 2024-03-29T10:43:53.128 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.143+0000 7fa501ac9640 10 monclient: build_initial_monmap 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.143+0000 7fa501ac9640 10 monclient: monmap: 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:43:53.131 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.143+0000 7fa501ac9640 1 librados: starting msgr at 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.143+0000 7fa501ac9640 1 librados: starting objecter 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.143+0000 7fa501ac9640 1 Processor -- start 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 1 -- start start 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 1 librados: setting wanted keys 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 1 librados: calling monclient init 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient: init 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:53.132 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:53.133 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient(hunting): picked mon.noname-c con 0x7fa4fc0b39f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:43:53.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient(hunting): picked mon.noname-a con 0x7fa4fc08b800 addr v1:172.21.15.94:6789/0 2024-03-29T10:43:53.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient(hunting): picked mon.noname-b con 0x7fa4fc141bd0 addr v1:172.21.15.173:6789/0 2024-03-29T10:43:53.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc140a80 con 0x7fa4fc08b800 2024-03-29T10:43:53.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc140fa0 con 0x7fa4fc141bd0 2024-03-29T10:43:53.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc0ae440 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:53.134 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.144+0000 7fa501ac9640 10 monclient(hunting): authenticate will time out at 1084.412980s 2024-03-29T10:43:53.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.152+0000 7fa4fbfff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fa4fc141bd0 0x7fa4fc0ad400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:34461/0 (socket says 172.21.15.94:34461) 2024-03-29T10:43:53.135 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.152+0000 7fa4fbfff640 1 -- 172.21.15.94:0/671329395 learned_addr learned my addr 172.21.15.94:0/671329395 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 client.?.objecter ms_handle_connect 0x7fa4fc141bd0 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 client.?.objecter ms_handle_connect 0x7fa4fc0b39f0 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328297903 0 0) 0x7fa4fc140fa0 con 0x7fa4fc141bd0 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e8004eb0 con 0x7fa4fc141bd0 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 client.?.objecter ms_handle_connect 0x7fa4fc08b800 2024-03-29T10:43:53.139 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724359866 0 0) 0x7fa4fc0ae440 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.153+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4fc140fa0 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2686095448 0 0) 0x7fa4e8004eb0 con 0x7fa4fc141bd0 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4fc0ae440 con 0x7fa4fc141bd0 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299235767 0 0) 0x7fa4fc140a80 con 0x7fa4fc08b800 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:53.140 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e8004eb0 con 0x7fa4fc08b800 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa4f4001f60 con 0x7fa4fc141bd0 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): dump: 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:53.141 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:53.142 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:53.142 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:53.142 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:53.143 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3628174407 0 0) 0x7fa4fc140fa0 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.143 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4fc140a80 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.143 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa4ec0018d0 con 0x7fa4fc0b39f0 2024-03-29T10:43:53.143 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:53.143 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:53.143 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.154+0000 7fa4e3fff640 10 monclient(hunting): dump: 2024-03-29T10:43:53.143 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1498666344 0 0) 0x7fa4fc0ae440 con 0x7fa4fc141bd0 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 >> v1:172.21.15.94:6790/0 conn(0x7fa4fc0b39f0 legacy=0x7fa4fc08b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 >> v1:172.21.15.94:6789/0 conn(0x7fa4fc08b800 legacy=0x7fa4fc1414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 1 monclient: found mon.b 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:53.144 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc142ea0 con 0x7fa4fc141bd0 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 5 monclient: authenticate success, global_id 4396 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 10 monclient: _renew_subs 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc070400 con 0x7fa4fc141bd0 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 10 monclient: _renew_subs 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc140430 con 0x7fa4fc141bd0 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa501ac9640 1 librados: init done 2024-03-29T10:43:53.145 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f40020c0 con 0x7fa4fc141bd0 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa4f4003dc0 con 0x7fa4fc141bd0 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.158+0000 7fa4e3fff640 10 monclient: dump: 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:43:53.146 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:43:53.147 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:43:53.147 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.159+0000 7fa4f97fa640 4 set_mon_vals no callback set 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.162+0000 7fa501ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.162+0000 7fa501ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.162+0000 7fa501ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.162+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc005080 con 0x7fa4fc141bd0 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.163+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fa4f4003f20 con 0x7fa4fc141bd0 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.164+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fa4f4084e80 con 0x7fa4fc141bd0 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.164+0000 7fa4e3fff640 10 client.4396.objecter ms_dispatch 0x7fa4fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.164+0000 7fa4e3fff640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.164+0000 7fa4e3fff640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.164+0000 7fa4e3fff640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.164+0000 7fa4fb7fe640 10 monclient: get_auth_request con 0x7fa4e806ddb0 auth_method 0 2024-03-29T10:43:53.148 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.164+0000 7fa4e3fff640 10 client.4396.objecter ms_handle_connect 0x7fa4e806ddb0 2024-03-29T10:43:53.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.171+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4f4050590 con 0x7fa4fc141bd0 2024-03-29T10:43:53.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.171+0000 7fa4e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:53.154 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.171+0000 7fa4e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:53.469 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.485+0000 7fa501ac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:53.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.485+0000 7fa501ac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:53.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.485+0000 7fa501ac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:43:53.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.485+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4dc005330 con 0x7fa4fc141bd0 2024-03-29T10:43:53.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.487+0000 7fa4e3fff640 1 -- 172.21.15.94:0/671329395 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fa4f4052690 con 0x7fa4fc141bd0 2024-03-29T10:43:53.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.487+0000 7fa4e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:53.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.487+0000 7fa4e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:53.471 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:43:53.471 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:53.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.490+0000 7fa501ac9640 10 librados: watch_flush enter 2024-03-29T10:43:53.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.490+0000 7fa501ac9640 10 librados: watch_flush exit 2024-03-29T10:43:53.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.490+0000 7fa501ac9640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-29T10:43:53.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.490+0000 7fa501ac9640 10 client.4396.objecter successfully canceled tick 2024-03-29T10:43:53.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.490+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 >> v1:172.21.15.94:6816/3829934808 conn(0x7fa4e806ddb0 legacy=0x7fa4e8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:53.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.491+0000 7fa501ac9640 10 monclient: shutdown 2024-03-29T10:43:53.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.491+0000 7fa501ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:53.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.491+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 >> v1:172.21.15.173:6789/0 conn(0x7fa4fc141bd0 legacy=0x7fa4fc0ad400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:53.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.493+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 shutdown_connections 2024-03-29T10:43:53.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.494+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 >> 172.21.15.94:0/671329395 conn(0x7fa4fc066a00 msgr2=0x7fa4fc0b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:53.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.494+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 shutdown_connections 2024-03-29T10:43:53.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.494+0000 7fa501ac9640 1 -- 172.21.15.94:0/671329395 wait complete. 2024-03-29T10:43:53.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:43:53.494+0000 7fa501ac9640 1 librados: shutdown 2024-03-29T10:43:53.496 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:00.303 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:00.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.468+0000 7f7a74f40640 10 monclient: get_monmap_and_config 2024-03-29T10:44:00.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.468+0000 7f7a74f40640 10 monclient: build_initial_monmap 2024-03-29T10:44:00.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.468+0000 7f7a74f40640 10 monclient: monmap: 2024-03-29T10:44:00.452 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:00.452 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:00.452 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:00.469527+0000 2024-03-29T10:44:00.452 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:00.469527+0000 2024-03-29T10:44:00.452 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:00.452 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:00.452 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:00.453 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:00.453 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:00.453 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:00.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.470+0000 7f7a74f40640 1 Processor -- start 2024-03-29T10:44:00.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.470+0000 7f7a74f40640 1 -- start start 2024-03-29T10:44:00.454 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.470+0000 7f7a74f40640 10 monclient: init 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 10 monclient(hunting): picked mon.noname-a con 0x7f7a700b0770 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 10 monclient(hunting): picked mon.noname-c con 0x7f7a7014a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 10 monclient(hunting): picked mon.noname-b con 0x7f7a700ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700aba70 con 0x7f7a700b0770 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700aecb0 con 0x7f7a7014a700 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a700af150 con 0x7f7a700ac5a0 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:00.457 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.471+0000 7f7a74f40640 10 monclient(hunting): authenticate will time out at 1091.739522s 2024-03-29T10:44:00.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.477+0000 7f7a6dd9b640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f7a7014a700 0x7f7a700abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:36605/0 (socket says 172.21.15.94:36605) 2024-03-29T10:44:00.460 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.477+0000 7f7a6dd9b640 1 -- 172.21.15.94:0/3456596058 learned_addr learned my addr 172.21.15.94:0/3456596058 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:00.463 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.480+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221793610 0 0) 0x7f7a700aba70 con 0x7f7a700b0770 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.480+0000 7f7a6d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.480+0000 7f7a6d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.480+0000 7f7a6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.480+0000 7f7a6d59a640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.480+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a58003520 con 0x7f7a700b0770 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154232755 0 0) 0x7f7a700af150 con 0x7f7a700ac5a0 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:44:00.464 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a700aba70 con 0x7f7a700ac5a0 2024-03-29T10:44:00.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922610328 0 0) 0x7f7a700aba70 con 0x7f7a700ac5a0 2024-03-29T10:44:00.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 >> v1:172.21.15.173:6789/0 conn(0x7f7a7014a700 legacy=0x7f7a700abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:00.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 >> v1:172.21.15.94:6789/0 conn(0x7f7a700b0770 legacy=0x7f7a70149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:00.465 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 monclient: found mon.noname-b 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a700ac9d0 con 0x7f7a700ac5a0 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient: _finish_auth 0 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a640015f0 con 0x7f7a700ac5a0 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.481+0000 7f7a6d59a640 10 monclient: dump: 2024-03-29T10:44:00.466 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.482+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a64001810 con 0x7f7a700ac5a0 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.482+0000 7f7a6d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:00.467 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.482+0000 7f7a6d59a640 1 -- 172.21.15.94:0/3456596058 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a64001ba0 con 0x7f7a700ac5a0 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.482+0000 7f7a6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.482+0000 7f7a6d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.482+0000 7f7a6d59a640 10 monclient: dump: 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:00.468 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:00.469 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:00.469 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:00.469 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:00.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.484+0000 7f7a74f40640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:44:00.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.484+0000 7f7a74f40640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:00.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.484+0000 7f7a74f40640 4 set_mon_vals no callback set 2024-03-29T10:44:00.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.485+0000 7f7a74f40640 10 monclient: shutdown 2024-03-29T10:44:00.470 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.485+0000 7f7a74f40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:00.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.485+0000 7f7a74f40640 1 -- 172.21.15.94:0/3456596058 >> v1:172.21.15.94:6790/0 conn(0x7f7a700ac5a0 legacy=0x7f7a70151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:00.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.486+0000 7f7a74f40640 1 -- 172.21.15.94:0/3456596058 shutdown_connections 2024-03-29T10:44:00.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.486+0000 7f7a74f40640 1 -- 172.21.15.94:0/3456596058 >> 172.21.15.94:0/3456596058 conn(0x7f7a7006b660 msgr2=0x7f7a7006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:00.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.486+0000 7f7a74f40640 1 -- 172.21.15.94:0/3456596058 shutdown_connections 2024-03-29T10:44:00.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.486+0000 7f7a74f40640 1 -- 172.21.15.94:0/3456596058 wait complete. 2024-03-29T10:44:00.471 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.487+0000 7f7a74f40640 10 monclient: build_initial_monmap 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.487+0000 7f7a74f40640 10 monclient: monmap: 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:00.472 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.488+0000 7f7a74f40640 1 librados: starting msgr at 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.488+0000 7f7a74f40640 1 librados: starting objecter 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.488+0000 7f7a74f40640 1 Processor -- start 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.488+0000 7f7a74f40640 1 -- start start 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.488+0000 7f7a74f40640 1 librados: setting wanted keys 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.488+0000 7f7a74f40640 1 librados: calling monclient init 2024-03-29T10:44:00.473 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.488+0000 7f7a74f40640 10 monclient: init 2024-03-29T10:44:00.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:00.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:00.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 10 monclient(hunting): picked mon.noname-c con 0x7f7a700ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:00.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 10 monclient(hunting): picked mon.noname-a con 0x7f7a700b0770 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:00.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 10 monclient(hunting): picked mon.noname-b con 0x7f7a7014a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:00.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a701e4550 con 0x7f7a700b0770 2024-03-29T10:44:00.475 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a701228a0 con 0x7f7a7014a700 2024-03-29T10:44:00.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a70123740 con 0x7f7a700ac5a0 2024-03-29T10:44:00.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:00.476 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.489+0000 7f7a74f40640 10 monclient(hunting): authenticate will time out at 1091.757855s 2024-03-29T10:44:00.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a6e59c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7a700ac5a0 0x7f7a70123030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:50099/0 (socket says 172.21.15.94:50099) 2024-03-29T10:44:00.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a6e59c640 1 -- 172.21.15.94:0/173410331 learned_addr learned my addr 172.21.15.94:0/173410331 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:00.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 10 client.?.objecter ms_handle_connect 0x7f7a700ac5a0 2024-03-29T10:44:00.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:00.477 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887484273 0 0) 0x7f7a70123740 con 0x7f7a700ac5a0 2024-03-29T10:44:00.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:00.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:00.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:00.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:44:00.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.493+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a50004eb0 con 0x7f7a700ac5a0 2024-03-29T10:44:00.478 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3438134213 0 0) 0x7f7a50004eb0 con 0x7f7a700ac5a0 2024-03-29T10:44:00.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a70123740 con 0x7f7a700ac5a0 2024-03-29T10:44:00.482 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a60001a10 con 0x7f7a700ac5a0 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 10 monclient(hunting): dump: 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.483 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2430101900 0 0) 0x7f7a70123740 con 0x7f7a700ac5a0 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 >> v1:172.21.15.173:6789/0 conn(0x7f7a7014a700 legacy=0x7f7a701ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.494+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 >> v1:172.21.15.94:6789/0 conn(0x7f7a700b0770 legacy=0x7f7a70120240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 1 monclient: found mon.c 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:00.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a701e49e0 con 0x7f7a700ac5a0 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 10 monclient: _finish_auth 0 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 5 monclient: authenticate success, global_id 4421 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 10 monclient: _renew_subs 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a701e4c50 con 0x7f7a700ac5a0 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:00.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 10 monclient: _renew_subs 2024-03-29T10:44:00.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:00.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a701e4060 con 0x7f7a700ac5a0 2024-03-29T10:44:00.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.498+0000 7f7a74f40640 1 librados: init done 2024-03-29T10:44:00.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.499+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a60001d10 con 0x7f7a700ac5a0 2024-03-29T10:44:00.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.499+0000 7f7a5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:00.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.499+0000 7f7a5ffff640 4 set_mon_vals no callback set 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7a60003a60 con 0x7f7a700ac5a0 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a5effd640 10 monclient: dump: 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.488 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a74f40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a74f40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:00.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a74f40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.500+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a54005080 con 0x7f7a700ac5a0 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.501+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f7a60003c80 con 0x7f7a700ac5a0 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.501+0000 7f7a6dd9b640 10 monclient: get_auth_request con 0x7f7a5006dbe0 auth_method 0 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.502+0000 7f7a5effd640 10 client.4421.objecter ms_handle_connect 0x7f7a5006dbe0 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.502+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f7a60084b60 con 0x7f7a700ac5a0 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.502+0000 7f7a5effd640 10 client.4421.objecter ms_dispatch 0x7f7a70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.502+0000 7f7a5effd640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.502+0000 7f7a5effd640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:00.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.502+0000 7f7a5effd640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T10:44:00.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.508+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a600504a0 con 0x7f7a700ac5a0 2024-03-29T10:44:00.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.508+0000 7f7a5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:00.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.508+0000 7f7a5effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:00.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.823+0000 7f7a74f40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:00.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.823+0000 7f7a74f40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:00.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.823+0000 7f7a74f40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:00.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.823+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a54005330 con 0x7f7a700ac5a0 2024-03-29T10:44:00.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.824+0000 7f7a5effd640 1 -- 172.21.15.94:0/173410331 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f7a60084210 con 0x7f7a700ac5a0 2024-03-29T10:44:00.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.824+0000 7f7a5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:00.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.824+0000 7f7a5effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:00.809 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:00.809 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:00.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.829+0000 7f7a74f40640 10 librados: watch_flush enter 2024-03-29T10:44:00.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.829+0000 7f7a74f40640 10 librados: watch_flush exit 2024-03-29T10:44:00.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.829+0000 7f7a74f40640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T10:44:00.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.829+0000 7f7a74f40640 10 client.4421.objecter successfully canceled tick 2024-03-29T10:44:00.816 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.832+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 >> v1:172.21.15.94:6816/3829934808 conn(0x7f7a5006dbe0 legacy=0x7f7a500700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:00.816 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.832+0000 7f7a74f40640 10 monclient: shutdown 2024-03-29T10:44:00.816 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.832+0000 7f7a74f40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:00.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.832+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 >> v1:172.21.15.94:6790/0 conn(0x7f7a700ac5a0 legacy=0x7f7a70123030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:00.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.833+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 shutdown_connections 2024-03-29T10:44:00.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.833+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 >> 172.21.15.94:0/173410331 conn(0x7f7a7006b660 msgr2=0x7f7a700b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:00.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.833+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 shutdown_connections 2024-03-29T10:44:00.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.833+0000 7f7a74f40640 1 -- 172.21.15.94:0/173410331 wait complete. 2024-03-29T10:44:00.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:00.833+0000 7f7a74f40640 1 librados: shutdown 2024-03-29T10:44:00.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:07.641 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:07.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.810+0000 7fb8ad372640 10 monclient: get_monmap_and_config 2024-03-29T10:44:07.794 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.810+0000 7fb8ad372640 10 monclient: build_initial_monmap 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.810+0000 7fb8ad372640 10 monclient: monmap: 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:07.811127+0000 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:07.811127+0000 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:07.795 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.811+0000 7fb8ad372640 1 Processor -- start 2024-03-29T10:44:07.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.811+0000 7fb8ad372640 1 -- start start 2024-03-29T10:44:07.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.811+0000 7fb8ad372640 10 monclient: init 2024-03-29T10:44:07.796 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.812+0000 7fb8ad372640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:07.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.812+0000 7fb8ad372640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:07.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8a6ffd640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fb8a80ae250 0x7fb8a813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:59375/0 (socket says 172.21.15.94:59375) 2024-03-29T10:44:07.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a80ae250 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:07.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a813f970 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:07.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a8142800 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:07.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a80afe00 con 0x7fb8a8142800 2024-03-29T10:44:07.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a8142c10 con 0x7fb8a813f970 2024-03-29T10:44:07.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a80afb20 con 0x7fb8a80ae250 2024-03-29T10:44:07.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:07.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8ad372640 10 monclient(hunting): authenticate will time out at 1099.084818s 2024-03-29T10:44:07.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8a6ffd640 1 -- 172.21.15.94:0/4281898084 learned_addr learned my addr 172.21.15.94:0/4281898084 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:07.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445567262 0 0) 0x7fb8a80afb20 con 0x7fb8a80ae250 2024-03-29T10:44:07.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:07.801 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:07.801 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:07.801 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.816+0000 7fb8a5ffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:44:07.801 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.817+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb890003520 con 0x7fb8a80ae250 2024-03-29T10:44:07.801 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.817+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1694351367 0 0) 0x7fb890003520 con 0x7fb8a80ae250 2024-03-29T10:44:07.801 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.817+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 >> v1:172.21.15.173:6789/0 conn(0x7fb8a813f970 legacy=0x7fb8a81420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:07.802 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 >> v1:172.21.15.94:6789/0 conn(0x7fb8a8142800 legacy=0x7fb8a814da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a80aab80 con 0x7fb8a80ae250 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:07.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb89c001690 con 0x7fb8a80ae250 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8a5ffb640 10 monclient: dump: 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:07.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8ad372640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:44:07.805 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.820+0000 7fb8ad372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:07.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb89c001920 con 0x7fb8a80ae250 2024-03-29T10:44:07.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8a5ffb640 1 -- 172.21.15.94:0/4281898084 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb89c001cb0 con 0x7fb8a80ae250 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8a5ffb640 10 monclient: dump: 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:07.807 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:07.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8ad372640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:07.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.823+0000 7fb8ad372640 4 set_mon_vals no callback set 2024-03-29T10:44:07.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.824+0000 7fb8ad372640 10 monclient: shutdown 2024-03-29T10:44:07.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.824+0000 7fb8ad372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:07.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.824+0000 7fb8ad372640 1 -- 172.21.15.94:0/4281898084 >> v1:172.21.15.94:6790/0 conn(0x7fb8a80ae250 legacy=0x7fb8a813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:07.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.826+0000 7fb8ad372640 1 -- 172.21.15.94:0/4281898084 shutdown_connections 2024-03-29T10:44:07.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.826+0000 7fb8ad372640 1 -- 172.21.15.94:0/4281898084 >> 172.21.15.94:0/4281898084 conn(0x7fb8a80b35e0 msgr2=0x7fb8a80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:07.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.828+0000 7fb8ad372640 1 -- 172.21.15.94:0/4281898084 shutdown_connections 2024-03-29T10:44:07.812 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.828+0000 7fb8ad372640 1 -- 172.21.15.94:0/4281898084 wait complete. 2024-03-29T10:44:07.812 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.829+0000 7fb8ad372640 10 monclient: build_initial_monmap 2024-03-29T10:44:07.812 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.829+0000 7fb8ad372640 10 monclient: monmap: 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.829+0000 7fb8ad372640 1 librados: starting msgr at 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.829+0000 7fb8ad372640 1 librados: starting objecter 2024-03-29T10:44:07.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.830+0000 7fb8ad372640 1 Processor -- start 2024-03-29T10:44:07.816 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.830+0000 7fb8ad372640 1 -- start start 2024-03-29T10:44:07.816 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.830+0000 7fb8ad372640 1 librados: setting wanted keys 2024-03-29T10:44:07.816 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.830+0000 7fb8ad372640 1 librados: calling monclient init 2024-03-29T10:44:07.816 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.830+0000 7fb8ad372640 10 monclient: init 2024-03-29T10:44:07.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.830+0000 7fb8ad372640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:07.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.831+0000 7fb8ad372640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:07.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a80ae250 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:07.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a813f970 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a8142800 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a8143a30 con 0x7fb8a8142800 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a813b960 con 0x7fb8a813f970 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a8139330 con 0x7fb8a80ae250 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8ad372640 10 monclient(hunting): authenticate will time out at 1099.103193s 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8a6ffd640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fb8a80ae250 0x7fb8a81e3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:59515/0 (socket says 172.21.15.94:59515) 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb8a6ffd640 1 -- 172.21.15.94:0/2743260678 learned_addr learned my addr 172.21.15.94:0/2743260678 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8a80ae250 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:07.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462048476 0 0) 0x7fb8a8139330 con 0x7fb8a80ae250 2024-03-29T10:44:07.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:07.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:07.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:07.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:44:07.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.835+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894004eb0 con 0x7fb8a80ae250 2024-03-29T10:44:07.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3791064024 0 0) 0x7fb894004eb0 con 0x7fb8a80ae250 2024-03-29T10:44:07.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8a8139330 con 0x7fb8a80ae250 2024-03-29T10:44:07.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb89c001610 con 0x7fb8a80ae250 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): dump: 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:07.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8a813f970 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485558724 0 0) 0x7fb8a813b960 con 0x7fb8a813f970 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.839+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894004eb0 con 0x7fb8a813f970 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.840+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 993798455 0 0) 0x7fb894004eb0 con 0x7fb8a813f970 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.840+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8a813b960 con 0x7fb8a813f970 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.840+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb8a0001b10 con 0x7fb8a813f970 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.840+0000 7fb88f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.840+0000 7fb88f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.840+0000 7fb88f7fe640 10 monclient(hunting): dump: 2024-03-29T10:44:07.828 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2892518405 0 0) 0x7fb8a813b960 con 0x7fb8a813f970 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 >> v1:172.21.15.94:6790/0 conn(0x7fb8a80ae250 legacy=0x7fb8a81e3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 >> v1:172.21.15.94:6789/0 conn(0x7fb8a8142800 legacy=0x7fb8a81ee8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:07.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 1 monclient: found mon.b 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a81394c0 con 0x7fb8a813f970 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb88f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 10 monclient: _renew_subs 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a813b3b0 con 0x7fb8a813f970 2024-03-29T10:44:07.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 10 monclient: _renew_subs 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a80ac3c0 con 0x7fb8a813f970 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.841+0000 7fb8ad372640 1 librados: init done 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.842+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a0002000 con 0x7fb8a813f970 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.842+0000 7fb88f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.842+0000 7fb8a4ff9640 4 set_mon_vals no callback set 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.842+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb8a0003bb0 con 0x7fb8a813f970 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.842+0000 7fb88f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.842+0000 7fb88f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.842+0000 7fb88f7fe640 10 monclient: dump: 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:07.831 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.843+0000 7fb8ad372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.843+0000 7fb8ad372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.843+0000 7fb8ad372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:07.832 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.843+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb888005080 con 0x7fb8a813f970 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.844+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fb8a0019b80 con 0x7fb8a813f970 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.845+0000 7fb8a6ffd640 10 monclient: get_auth_request con 0x7fb89406df80 auth_method 0 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.846+0000 7fb88f7fe640 10 client.4402.objecter ms_handle_connect 0x7fb89406df80 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.846+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fb8a0083ab0 con 0x7fb8a813f970 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.846+0000 7fb88f7fe640 10 client.4402.objecter ms_dispatch 0x7fb8a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.846+0000 7fb88f7fe640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.846+0000 7fb88f7fe640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:07.834 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.846+0000 7fb88f7fe640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T10:44:07.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.851+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8a004f390 con 0x7fb8a813f970 2024-03-29T10:44:07.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.851+0000 7fb88f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:07.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:07.851+0000 7fb88f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:08.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.179+0000 7fb8ad372640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:08.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.179+0000 7fb8ad372640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:08.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.179+0000 7fb8ad372640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:08.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.179+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb888005330 con 0x7fb8a813f970 2024-03-29T10:44:08.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.181+0000 7fb88f7fe640 1 -- 172.21.15.94:0/2743260678 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fb8a00512c0 con 0x7fb8a813f970 2024-03-29T10:44:08.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.181+0000 7fb88f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:08.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.181+0000 7fb88f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:08.165 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:08.165 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:08.168 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.185+0000 7fb8ad372640 10 librados: watch_flush enter 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.185+0000 7fb8ad372640 10 librados: watch_flush exit 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.185+0000 7fb8ad372640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.185+0000 7fb8ad372640 10 client.4402.objecter successfully canceled tick 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.186+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 >> v1:172.21.15.94:6816/3829934808 conn(0x7fb89406df80 legacy=0x7fb894070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.186+0000 7fb8ad372640 10 monclient: shutdown 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.186+0000 7fb8ad372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.186+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 >> v1:172.21.15.173:6789/0 conn(0x7fb8a813f970 legacy=0x7fb8a81e45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.186+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 shutdown_connections 2024-03-29T10:44:08.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.186+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 >> 172.21.15.94:0/2743260678 conn(0x7fb8a80b35e0 msgr2=0x7fb8a80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:08.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.186+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 shutdown_connections 2024-03-29T10:44:08.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.187+0000 7fb8ad372640 1 -- 172.21.15.94:0/2743260678 wait complete. 2024-03-29T10:44:08.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:08.187+0000 7fb8ad372640 1 librados: shutdown 2024-03-29T10:44:08.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:14.993 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:15.149 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.165+0000 7fc58d146640 10 monclient: get_monmap_and_config 2024-03-29T10:44:15.149 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.165+0000 7fc58d146640 10 monclient: build_initial_monmap 2024-03-29T10:44:15.149 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.165+0000 7fc58d146640 10 monclient: monmap: 2024-03-29T10:44:15.149 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:15.149 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:15.149 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:15.166376+0000 2024-03-29T10:44:15.150 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:15.166376+0000 2024-03-29T10:44:15.150 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:15.150 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:15.150 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:15.150 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:15.150 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:15.150 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:15.151 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.168+0000 7fc58d146640 1 Processor -- start 2024-03-29T10:44:15.151 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.168+0000 7fc58d146640 1 -- start start 2024-03-29T10:44:15.151 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.168+0000 7fc58d146640 10 monclient: init 2024-03-29T10:44:15.152 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.169+0000 7fc58d146640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:15.152 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.169+0000 7fc58d146640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.172+0000 7fc58d146640 10 monclient(hunting): picked mon.noname-a con 0x7fc580034850 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.172+0000 7fc58d146640 10 monclient(hunting): picked mon.noname-c con 0x7fc5800c9550 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.172+0000 7fc58d146640 10 monclient(hunting): picked mon.noname-b con 0x7fc58002aac0 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.173+0000 7fc58d146640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58002c440 con 0x7fc580034850 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.173+0000 7fc587fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc580034850 0x7fc5800c8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52013/0 (socket says 172.21.15.94:52013) 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.173+0000 7fc58d146640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58002aed0 con 0x7fc5800c9550 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.173+0000 7fc587fff640 1 -- 172.21.15.94:0/3097085420 learned_addr learned my addr 172.21.15.94:0/3097085420 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.173+0000 7fc58d146640 1 -- 172.21.15.94:0/3097085420 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58002b190 con 0x7fc58002aac0 2024-03-29T10:44:15.158 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.173+0000 7fc58d146640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.173+0000 7fc58d146640 10 monclient(hunting): authenticate will time out at 1106.441989s 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904499259 0 0) 0x7fc58002c440 con 0x7fc580034850 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc578003520 con 0x7fc580034850 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 133437656 0 0) 0x7fc578003520 con 0x7fc580034850 2024-03-29T10:44:15.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.174+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 >> v1:172.21.15.94:6790/0 conn(0x7fc58002aac0 legacy=0x7fc5800cc5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.177+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 >> v1:172.21.15.173:6789/0 conn(0x7fc5800c9550 legacy=0x7fc5800cbea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.178+0000 7fc586ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 1 monclient: found mon.noname-a 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc580033060 con 0x7fc580034850 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc57c001690 con 0x7fc580034850 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: dump: 2024-03-29T10:44:15.164 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc58d146640 5 monclient: authenticate success, global_id 4413 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc58d146640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc57c001860 con 0x7fc580034850 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 1 -- 172.21.15.94:0/3097085420 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc57c001cb0 con 0x7fc580034850 2024-03-29T10:44:15.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc586ffd640 10 monclient: dump: 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc58d146640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:15.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.179+0000 7fc58d146640 4 set_mon_vals no callback set 2024-03-29T10:44:15.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.180+0000 7fc58d146640 10 monclient: shutdown 2024-03-29T10:44:15.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.180+0000 7fc58d146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:15.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.180+0000 7fc58d146640 1 -- 172.21.15.94:0/3097085420 >> v1:172.21.15.94:6789/0 conn(0x7fc580034850 legacy=0x7fc5800c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:15.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.184+0000 7fc58d146640 1 -- 172.21.15.94:0/3097085420 shutdown_connections 2024-03-29T10:44:15.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.184+0000 7fc58d146640 1 -- 172.21.15.94:0/3097085420 >> 172.21.15.94:0/3097085420 conn(0x7fc5800058f0 msgr2=0x7fc580005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:15.167 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.184+0000 7fc58d146640 1 -- 172.21.15.94:0/3097085420 shutdown_connections 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.185+0000 7fc58d146640 1 -- 172.21.15.94:0/3097085420 wait complete. 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.185+0000 7fc58d146640 10 monclient: build_initial_monmap 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.185+0000 7fc58d146640 10 monclient: monmap: 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:15.171 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.186+0000 7fc58d146640 1 librados: starting msgr at 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.186+0000 7fc58d146640 1 librados: starting objecter 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.186+0000 7fc58d146640 1 Processor -- start 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.186+0000 7fc58d146640 1 -- start start 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.186+0000 7fc58d146640 1 librados: setting wanted keys 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.186+0000 7fc58d146640 1 librados: calling monclient init 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.186+0000 7fc58d146640 10 monclient: init 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.187+0000 7fc58d146640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:15.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.187+0000 7fc58d146640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 10 monclient(hunting): picked mon.noname-a con 0x7fc58002aac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 10 monclient(hunting): picked mon.noname-b con 0x7fc580034850 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 10 monclient(hunting): picked mon.noname-c con 0x7fc5800c9550 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800bee60 con 0x7fc58002aac0 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800c13f0 con 0x7fc580034850 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800c16d0 con 0x7fc5800c9550 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:15.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.189+0000 7fc58d146640 10 monclient(hunting): authenticate will time out at 1106.457464s 2024-03-29T10:44:15.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.190+0000 7fc587fff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc58002aac0 0x7fc5800bf680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:52227/0 (socket says 172.21.15.94:52227) 2024-03-29T10:44:15.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.190+0000 7fc587fff640 1 -- 172.21.15.94:0/1191756040 learned_addr learned my addr 172.21.15.94:0/1191756040 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:15.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.190+0000 7fc584ff9640 10 client.?.objecter ms_handle_connect 0x7fc58002aac0 2024-03-29T10:44:15.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.190+0000 7fc584ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231250228 0 0) 0x7fc5800bee60 con 0x7fc58002aac0 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc56c004eb0 con 0x7fc58002aac0 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 285419517 0 0) 0x7fc56c004eb0 con 0x7fc58002aac0 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5800bee60 con 0x7fc58002aac0 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc57c001b70 con 0x7fc58002aac0 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:15.178 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.191+0000 7fc584ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.193+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1651326969 0 0) 0x7fc5800bee60 con 0x7fc58002aac0 2024-03-29T10:44:15.179 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.193+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 >> v1:172.21.15.94:6790/0 conn(0x7fc5800c9550 legacy=0x7fc5800bc8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:15.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 >> v1:172.21.15.173:6789/0 conn(0x7fc580034850 legacy=0x7fc580169d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:44:15.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:15.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 1 monclient: found mon.a 2024-03-29T10:44:15.180 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:15.181 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:15.181 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5800c5150 con 0x7fc58002aac0 2024-03-29T10:44:15.181 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc57c004bf0 con 0x7fc58002aac0 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc584ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.197+0000 7fc5867fc640 4 set_mon_vals no callback set 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.198+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc57c005040 con 0x7fc58002aac0 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.198+0000 7fc584ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.198+0000 7fc584ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.198+0000 7fc584ff9640 10 monclient: dump: 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.183 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 5 monclient: authenticate success, global_id 4416 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 10 monclient: _renew_subs 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58016a470 con 0x7fc58002aac0 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 10 monclient: _renew_subs 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5800bd1b0 con 0x7fc58002aac0 2024-03-29T10:44:15.184 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.199+0000 7fc58d146640 1 librados: init done 2024-03-29T10:44:15.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.201+0000 7fc58d146640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:15.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.201+0000 7fc58d146640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:15.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.201+0000 7fc58d146640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:15.185 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.201+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5800beff0 con 0x7fc58002aac0 2024-03-29T10:44:15.190 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.208+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fc57c01b010 con 0x7fc58002aac0 2024-03-29T10:44:15.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.209+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fc57c084c60 con 0x7fc58002aac0 2024-03-29T10:44:15.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.209+0000 7fc584ff9640 10 client.4416.objecter ms_dispatch 0x7fc588000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:15.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.209+0000 7fc584ff9640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:15.191 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.209+0000 7fc584ff9640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:15.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.209+0000 7fc584ff9640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T10:44:15.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.209+0000 7fc5877fe640 10 monclient: get_auth_request con 0x7fc56c06dc30 auth_method 0 2024-03-29T10:44:15.192 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.209+0000 7fc584ff9640 10 client.4416.objecter ms_handle_connect 0x7fc56c06dc30 2024-03-29T10:44:15.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.213+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc57c050540 con 0x7fc58002aac0 2024-03-29T10:44:15.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.213+0000 7fc584ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:15.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.213+0000 7fc584ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:15.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.532+0000 7fc58d146640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:15.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.532+0000 7fc58d146640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:15.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.532+0000 7fc58d146640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:15.516 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.532+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5800c52e0 con 0x7fc58002aac0 2024-03-29T10:44:15.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.536+0000 7fc584ff9640 1 -- 172.21.15.94:0/1191756040 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fc57c052470 con 0x7fc58002aac0 2024-03-29T10:44:15.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.536+0000 7fc584ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:15.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.536+0000 7fc584ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:15.519 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:15.519 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.539+0000 7fc58d146640 10 librados: watch_flush enter 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 10 librados: watch_flush exit 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 10 client.4416.objecter successfully canceled tick 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 >> v1:172.21.15.94:6816/3829934808 conn(0x7fc56c06dc30 legacy=0x7fc56c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 10 monclient: shutdown 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 >> v1:172.21.15.94:6789/0 conn(0x7fc58002aac0 legacy=0x7fc5800bf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 shutdown_connections 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 >> 172.21.15.94:0/1191756040 conn(0x7fc5800058f0 msgr2=0x7fc5800043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 shutdown_connections 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 1 -- 172.21.15.94:0/1191756040 wait complete. 2024-03-29T10:44:15.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:15.540+0000 7fc58d146640 1 librados: shutdown 2024-03-29T10:44:15.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:22.345 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:22.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.504+0000 7fb18d94d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.504+0000 7fb18d94d640 10 monclient: build_initial_monmap 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.505+0000 7fb18d94d640 10 monclient: monmap: 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:22.505853+0000 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:22.505853+0000 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:22.488 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:22.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.505+0000 7fb18d94d640 1 Processor -- start 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 1 -- start start 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient: init 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient(hunting): picked mon.noname-a con 0x7fb1880b0770 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient(hunting): picked mon.noname-c con 0x7fb18814a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient(hunting): picked mon.noname-b con 0x7fb1880ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880aba70 con 0x7fb1880b0770 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880aecb0 con 0x7fb18814a700 2024-03-29T10:44:22.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880af150 con 0x7fb1880ac5a0 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.506+0000 7fb18d94d640 10 monclient(hunting): authenticate will time out at 1113.775039s 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb1867fc640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fb18814a700 0x7fb1880abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:59846/0 (socket says 172.21.15.94:59846) 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb1867fc640 1 -- 172.21.15.94:0/2671676131 learned_addr learned my addr 172.21.15.94:0/2671676131 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276248337 0 0) 0x7fb1880aecb0 con 0x7fb18814a700 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb174003520 con 0x7fb18814a700 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438910778 0 0) 0x7fb1880af150 con 0x7fb1880ac5a0 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:22.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:44:22.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.507+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1880aecb0 con 0x7fb1880ac5a0 2024-03-29T10:44:22.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.508+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 901176901 0 0) 0x7fb174003520 con 0x7fb18814a700 2024-03-29T10:44:22.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.508+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 >> v1:172.21.15.94:6790/0 conn(0x7fb1880ac5a0 legacy=0x7fb188151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:22.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.508+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 >> v1:172.21.15.94:6789/0 conn(0x7fb1880b0770 legacy=0x7fb188149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:22.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:22.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 1 monclient: found mon.noname-c 2024-03-29T10:44:22.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:22.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:44:22.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1880ac9d0 con 0x7fb18814a700 2024-03-29T10:44:22.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb180001a30 con 0x7fb18814a700 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: dump: 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:22.495 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb18d94d640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb18d94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb180001c50 con 0x7fb18814a700 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 1 -- 172.21.15.94:0/2671676131 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb180001fe0 con 0x7fb18814a700 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb185ffb640 10 monclient: dump: 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:22.496 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb18d94d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.509+0000 7fb18d94d640 4 set_mon_vals no callback set 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.510+0000 7fb18d94d640 10 monclient: shutdown 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.510+0000 7fb18d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.510+0000 7fb18d94d640 1 -- 172.21.15.94:0/2671676131 >> v1:172.21.15.173:6789/0 conn(0x7fb18814a700 legacy=0x7fb1880abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:22.497 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.510+0000 7fb18d94d640 1 -- 172.21.15.94:0/2671676131 shutdown_connections 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.510+0000 7fb18d94d640 1 -- 172.21.15.94:0/2671676131 >> 172.21.15.94:0/2671676131 conn(0x7fb18806b660 msgr2=0x7fb18806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.511+0000 7fb18d94d640 1 -- 172.21.15.94:0/2671676131 shutdown_connections 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.511+0000 7fb18d94d640 1 -- 172.21.15.94:0/2671676131 wait complete. 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.511+0000 7fb18d94d640 10 monclient: build_initial_monmap 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.511+0000 7fb18d94d640 10 monclient: monmap: 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:22.498 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:22.499 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:22.499 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:22.499 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:22.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.512+0000 7fb18d94d640 1 librados: starting msgr at 2024-03-29T10:44:22.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.512+0000 7fb18d94d640 1 librados: starting objecter 2024-03-29T10:44:22.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.512+0000 7fb18d94d640 1 Processor -- start 2024-03-29T10:44:22.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.512+0000 7fb18d94d640 1 -- start start 2024-03-29T10:44:22.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.512+0000 7fb18d94d640 1 librados: setting wanted keys 2024-03-29T10:44:22.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.512+0000 7fb18d94d640 1 librados: calling monclient init 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.512+0000 7fb18d94d640 10 monclient: init 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 10 monclient(hunting): picked mon.noname-a con 0x7fb1880ac5a0 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 10 monclient(hunting): picked mon.noname-b con 0x7fb1880b0770 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 10 monclient(hunting): picked mon.noname-c con 0x7fb18814a700 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1881239d0 con 0x7fb1880ac5a0 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb188128350 con 0x7fb1880b0770 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb188125fd0 con 0x7fb18814a700 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb18d94d640 10 monclient(hunting): authenticate will time out at 1113.781740s 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb1867fc640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fb1880b0770 0x7fb1881dbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:60338/0 (socket says 172.21.15.94:60338) 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.513+0000 7fb1867fc640 1 -- 172.21.15.94:0/2596268179 learned_addr learned my addr 172.21.15.94:0/2596268179 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 client.?.objecter ms_handle_connect 0x7fb1880b0770 2024-03-29T10:44:22.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687744177 0 0) 0x7fb188128350 con 0x7fb1880b0770 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb168004eb0 con 0x7fb1880b0770 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 client.?.objecter ms_handle_connect 0x7fb18814a700 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736417269 0 0) 0x7fb168004eb0 con 0x7fb1880b0770 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb188128350 con 0x7fb1880b0770 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb180001fa0 con 0x7fb1880b0770 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:22.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): dump: 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787402155 0 0) 0x7fb188125fd0 con 0x7fb18814a700 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:22.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb168004eb0 con 0x7fb18814a700 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.514+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 975313423 0 0) 0x7fb188128350 con 0x7fb1880b0770 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.515+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 >> v1:172.21.15.94:6790/0 conn(0x7fb18814a700 legacy=0x7fb1881e7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.515+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 >> v1:172.21.15.94:6789/0 conn(0x7fb1880ac5a0 legacy=0x7fb188144fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 1 monclient: found mon.b 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb188126f60 con 0x7fb1880b0770 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 10 monclient: _finish_auth 0 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb17ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:22.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 5 monclient: authenticate success, global_id 4408 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 10 monclient: _renew_subs 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1881273c0 con 0x7fb1880b0770 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 10 monclient: _renew_subs 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1881458c0 con 0x7fb1880b0770 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.517+0000 7fb18d94d640 1 librados: init done 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb180004e10 con 0x7fb1880b0770 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb17ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb1800015f0 con 0x7fb1880b0770 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb17ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb17ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:22.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb17ffff640 10 monclient: dump: 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:22.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb1857fa640 4 set_mon_vals no callback set 2024-03-29T10:44:22.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.518+0000 7fb18d94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:22.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.519+0000 7fb18d94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:22.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.519+0000 7fb18d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:22.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.519+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16c005080 con 0x7fb1880b0770 2024-03-29T10:44:22.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.520+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fb180005260 con 0x7fb1880b0770 2024-03-29T10:44:22.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.520+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fb180084ed0 con 0x7fb1880b0770 2024-03-29T10:44:22.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.520+0000 7fb17ffff640 10 client.4408.objecter ms_dispatch 0x7fb188000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.520+0000 7fb17ffff640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.520+0000 7fb17ffff640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.521+0000 7fb186ffd640 10 monclient: get_auth_request con 0x7fb16806dc40 auth_method 0 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.521+0000 7fb17ffff640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.521+0000 7fb17ffff640 10 client.4408.objecter ms_handle_connect 0x7fb16806dc40 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.527+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1800507b0 con 0x7fb1880b0770 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.527+0000 7fb17ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:22.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.527+0000 7fb17ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:22.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.839+0000 7fb18d94d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:22.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.839+0000 7fb18d94d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:22.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.839+0000 7fb18d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:22.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.839+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb16c005330 con 0x7fb1880b0770 2024-03-29T10:44:22.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.840+0000 7fb17ffff640 1 -- 172.21.15.94:0/2596268179 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fb1800526e0 con 0x7fb1880b0770 2024-03-29T10:44:22.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.840+0000 7fb17ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:22.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.840+0000 7fb17ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:22.823 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:22.823 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.843+0000 7fb18d94d640 10 librados: watch_flush enter 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.843+0000 7fb18d94d640 10 librados: watch_flush exit 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 10 client.4408.objecter successfully canceled tick 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 >> v1:172.21.15.94:6816/3829934808 conn(0x7fb16806dc40 legacy=0x7fb168070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 10 monclient: shutdown 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 >> v1:172.21.15.173:6789/0 conn(0x7fb1880b0770 legacy=0x7fb1881dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 shutdown_connections 2024-03-29T10:44:22.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 >> 172.21.15.94:0/2596268179 conn(0x7fb18806b660 msgr2=0x7fb188071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:22.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 shutdown_connections 2024-03-29T10:44:22.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 1 -- 172.21.15.94:0/2596268179 wait complete. 2024-03-29T10:44:22.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:22.844+0000 7fb18d94d640 1 librados: shutdown 2024-03-29T10:44:22.844 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:29.652 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:29.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.836+0000 7f54a72ba640 10 monclient: get_monmap_and_config 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.836+0000 7f54a72ba640 10 monclient: build_initial_monmap 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.836+0000 7f54a72ba640 10 monclient: monmap: 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:29.837771+0000 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:29.837771+0000 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:29.820 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:29.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.838+0000 7f54a72ba640 1 Processor -- start 2024-03-29T10:44:29.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.838+0000 7f54a72ba640 1 -- start start 2024-03-29T10:44:29.821 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.838+0000 7f54a72ba640 10 monclient: init 2024-03-29T10:44:29.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:29.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:29.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 10 monclient(hunting): picked mon.noname-c con 0x7f54a00b4710 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:29.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 10 monclient(hunting): picked mon.noname-a con 0x7f54a00b3970 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:29.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 10 monclient(hunting): picked mon.noname-b con 0x7f54a013f390 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:29.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a00af020 con 0x7f54a00b3970 2024-03-29T10:44:29.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a00b0910 con 0x7f54a00b4710 2024-03-29T10:44:29.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a00b0bf0 con 0x7f54a013f390 2024-03-29T10:44:29.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:29.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a72ba640 10 monclient(hunting): authenticate will time out at 1121.107703s 2024-03-29T10:44:29.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a5056640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f54a00b4710 0x7f54a00b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45766/0 (socket says 172.21.15.94:45766) 2024-03-29T10:44:29.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.839+0000 7f54a5056640 1 -- 172.21.15.94:0/2001618514 learned_addr learned my addr 172.21.15.94:0/2001618514 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440467330 0 0) 0x7f54a00b0bf0 con 0x7f54a013f390 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c003520 con 0x7f54a013f390 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240641242 0 0) 0x7f54a00b0910 con 0x7f54a00b4710 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:29.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a00b0bf0 con 0x7f54a00b4710 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820512578 0 0) 0x7f54a00af020 con 0x7f54a00b3970 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.840+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a00b0910 con 0x7f54a00b3970 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2507322107 0 0) 0x7f54a00b0bf0 con 0x7f54a00b4710 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 >> v1:172.21.15.94:6790/0 conn(0x7f54a013f390 legacy=0x7f54a013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 >> v1:172.21.15.94:6789/0 conn(0x7f54a00b3970 legacy=0x7f54a00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 1 monclient: found mon.noname-c 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:44:29.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a00aeb00 con 0x7f54a00b4710 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f54900015a0 con 0x7f54a00b4710 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f5497fff640 10 monclient: dump: 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:29.826 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:29.827 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:29.827 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:29.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f54a72ba640 5 monclient: authenticate success, global_id 4411 2024-03-29T10:44:29.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.841+0000 7f54a72ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5490001700 con 0x7f54a00b4710 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f5497fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f5497fff640 1 -- 172.21.15.94:0/2001618514 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5490001b10 con 0x7f54a00b4710 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f54a72ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f5497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f54a72ba640 4 set_mon_vals no callback set 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f5497fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:29.828 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.842+0000 7f5497fff640 10 monclient: dump: 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.843+0000 7f54a72ba640 10 monclient: shutdown 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.843+0000 7f54a72ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.843+0000 7f54a72ba640 1 -- 172.21.15.94:0/2001618514 >> v1:172.21.15.173:6789/0 conn(0x7f54a00b4710 legacy=0x7f54a00b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.843+0000 7f54a72ba640 1 -- 172.21.15.94:0/2001618514 shutdown_connections 2024-03-29T10:44:29.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.843+0000 7f54a72ba640 1 -- 172.21.15.94:0/2001618514 >> 172.21.15.94:0/2001618514 conn(0x7f54a0066a00 msgr2=0x7f54a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.843+0000 7f54a72ba640 1 -- 172.21.15.94:0/2001618514 shutdown_connections 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.843+0000 7f54a72ba640 1 -- 172.21.15.94:0/2001618514 wait complete. 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.844+0000 7f54a72ba640 10 monclient: build_initial_monmap 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.844+0000 7f54a72ba640 10 monclient: monmap: 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:29.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.845+0000 7f54a72ba640 1 librados: starting msgr at 2024-03-29T10:44:29.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.845+0000 7f54a72ba640 1 librados: starting objecter 2024-03-29T10:44:29.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.845+0000 7f54a72ba640 1 Processor -- start 2024-03-29T10:44:29.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.845+0000 7f54a72ba640 1 -- start start 2024-03-29T10:44:29.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.845+0000 7f54a72ba640 1 librados: setting wanted keys 2024-03-29T10:44:29.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.845+0000 7f54a72ba640 1 librados: calling monclient init 2024-03-29T10:44:29.831 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.845+0000 7f54a72ba640 10 monclient: init 2024-03-29T10:44:29.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f54a72ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:29.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f54a72ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:29.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f54a72ba640 10 monclient(hunting): picked mon.noname-b con 0x7f54a00b3970 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f54a72ba640 10 monclient(hunting): picked mon.noname-c con 0x7f54a013f390 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f54a4855640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f54a013f390 0x7f54a0135b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:52622/0 (socket says 172.21.15.94:52622) 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f54a4855640 1 -- 172.21.15.94:0/1393462839 learned_addr learned my addr 172.21.15.94:0/1393462839 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f5495ffb640 10 client.?.objecter ms_handle_connect 0x7f54a013f390 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.846+0000 7f54a5056640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f54a00b3970 0x7f54a0137700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45950/0 (socket says 172.21.15.94:45950) 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f54a72ba640 10 monclient(hunting): picked mon.noname-a con 0x7f54a0141ba0 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a01356d0 con 0x7f54a0141ba0 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a0136d20 con 0x7f54a00b3970 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a0137000 con 0x7f54a013f390 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f54a72ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f54a72ba640 10 monclient(hunting): authenticate will time out at 1121.115306s 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 client.?.objecter ms_handle_connect 0x7f54a00b3970 2024-03-29T10:44:29.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 client.?.objecter ms_handle_connect 0x7f54a0141ba0 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170405615 0 0) 0x7f54a0136d20 con 0x7f54a00b3970 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.847+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5484004eb0 con 0x7f54a00b3970 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382858215 0 0) 0x7f54a01356d0 con 0x7f54a0141ba0 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a0136d20 con 0x7f54a0141ba0 2024-03-29T10:44:29.837 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242759337 0 0) 0x7f54a0137000 con 0x7f54a013f390 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a01356d0 con 0x7f54a013f390 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077829475 0 0) 0x7f5484004eb0 con 0x7f54a00b3970 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54a0137000 con 0x7f54a00b3970 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5490001a50 con 0x7f54a00b3970 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.838 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.848+0000 7f5495ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3883959307 0 0) 0x7f54a0137000 con 0x7f54a00b3970 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 >> v1:172.21.15.94:6790/0 conn(0x7f54a013f390 legacy=0x7f54a0135b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 >> v1:172.21.15.94:6789/0 conn(0x7f54a0141ba0 legacy=0x7f54a01e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 1 monclient: found mon.b 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:29.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a01372e0 con 0x7f54a00b3970 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f5495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 5 monclient: authenticate success, global_id 4414 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 10 monclient: _renew_subs 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a0134c80 con 0x7f54a00b3970 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 10 monclient: _renew_subs 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.849+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a01e3710 con 0x7f54a00b3970 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.850+0000 7f54a72ba640 1 librados: init done 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.850+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5490004a90 con 0x7f54a00b3970 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.850+0000 7f5495ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:29.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.850+0000 7f54977fe640 4 set_mon_vals no callback set 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.851+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5490004da0 con 0x7f54a00b3970 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.851+0000 7f5495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.851+0000 7f5495ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.851+0000 7f5495ffb640 10 monclient: dump: 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:29.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.852+0000 7f54a72ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:29.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.852+0000 7f54a72ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:29.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.852+0000 7f54a72ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:29.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.852+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a00688a0 con 0x7f54a00b3970 2024-03-29T10:44:29.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.853+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f549001ad70 con 0x7f54a00b3970 2024-03-29T10:44:29.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.854+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f5490084a50 con 0x7f54a00b3970 2024-03-29T10:44:29.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.854+0000 7f5495ffb640 10 client.4414.objecter ms_dispatch 0x7f54a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:29.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.854+0000 7f5495ffb640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:29.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.854+0000 7f54a4855640 10 monclient: get_auth_request con 0x7f548406dc90 auth_method 0 2024-03-29T10:44:29.843 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.854+0000 7f5495ffb640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:29.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.854+0000 7f5495ffb640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T10:44:29.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.855+0000 7f5495ffb640 10 client.4414.objecter ms_handle_connect 0x7f548406dc90 2024-03-29T10:44:29.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.860+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5490050330 con 0x7f54a00b3970 2024-03-29T10:44:29.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.860+0000 7f5495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:29.844 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:29.860+0000 7f5495ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:30.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.233+0000 7f54a72ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:30.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.233+0000 7f54a72ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:30.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.233+0000 7f54a72ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:44:30.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.233+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a0135440 con 0x7f54a00b3970 2024-03-29T10:44:30.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.234+0000 7f5495ffb640 1 -- 172.21.15.94:0/1393462839 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f5490052260 con 0x7f54a00b3970 2024-03-29T10:44:30.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.234+0000 7f5495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:30.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.234+0000 7f5495ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:30.217 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:30.217 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:30.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 10 librados: watch_flush enter 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 10 librados: watch_flush exit 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 10 client.4414.objecter successfully canceled tick 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 >> v1:172.21.15.94:6816/3829934808 conn(0x7f548406dc90 legacy=0x7f5484070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 10 monclient: shutdown 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 >> v1:172.21.15.173:6789/0 conn(0x7f54a00b3970 legacy=0x7f54a0137700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 shutdown_connections 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 >> 172.21.15.94:0/1393462839 conn(0x7f54a0066a00 msgr2=0x7f54a0139070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 shutdown_connections 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.238+0000 7f54a72ba640 1 -- 172.21.15.94:0/1393462839 wait complete. 2024-03-29T10:44:30.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:30.239+0000 7f54a72ba640 1 librados: shutdown 2024-03-29T10:44:30.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:37.044 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:37.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.211+0000 7f7df1a9a640 10 monclient: get_monmap_and_config 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.211+0000 7f7df1a9a640 10 monclient: build_initial_monmap 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.211+0000 7f7df1a9a640 10 monclient: monmap: 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:37.212984+0000 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:37.212984+0000 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:37.195 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:37.196 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.212+0000 7f7df1a9a640 1 Processor -- start 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.212+0000 7f7df1a9a640 1 -- start start 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.212+0000 7f7df1a9a640 10 monclient: init 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec06f050 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec070240 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec06f430 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec13f400 con 0x7f7dec06f430 2024-03-29T10:44:37.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec1429c0 con 0x7f7dec070240 2024-03-29T10:44:37.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec06ff80 con 0x7f7dec06f050 2024-03-29T10:44:37.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:37.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7df1a9a640 10 monclient(hunting): authenticate will time out at 1128.482616s 2024-03-29T10:44:37.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7debfff640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7dec06f430 0x7f7dec14a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42734/0 (socket says 172.21.15.94:42734) 2024-03-29T10:44:37.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.213+0000 7f7debfff640 1 -- 172.21.15.94:0/2568730612 learned_addr learned my addr 172.21.15.94:0/2568730612 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:37.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295287113 0 0) 0x7f7dec13f400 con 0x7f7dec06f430 2024-03-29T10:44:37.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.199 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd4003520 con 0x7f7dec06f430 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321739909 0 0) 0x7f7dec06ff80 con 0x7f7dec06f050 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec13f400 con 0x7f7dec06f050 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196819963 0 0) 0x7f7dd4003520 con 0x7f7dec06f430 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 >> v1:172.21.15.94:6790/0 conn(0x7f7dec06f050 legacy=0x7f7dec1422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 >> v1:172.21.15.173:6789/0 conn(0x7f7dec070240 legacy=0x7f7dec0ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 monclient: found mon.noname-a 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:37.200 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec0aef70 con 0x7f7dec06f430 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7de4001690 con 0x7f7dec06f430 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: dump: 2024-03-29T10:44:37.201 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7df1a9a640 5 monclient: authenticate success, global_id 4425 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7df1a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7de40017f0 con 0x7f7dec06f430 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:37.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 1 -- 172.21.15.94:0/2568730612 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7de4001c00 con 0x7f7dec06f430 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7dea7fc640 10 monclient: dump: 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7df1a9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:37.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.214+0000 7f7df1a9a640 4 set_mon_vals no callback set 2024-03-29T10:44:37.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.215+0000 7f7df1a9a640 10 monclient: shutdown 2024-03-29T10:44:37.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.215+0000 7f7df1a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:37.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.215+0000 7f7df1a9a640 1 -- 172.21.15.94:0/2568730612 >> v1:172.21.15.94:6789/0 conn(0x7f7dec06f430 legacy=0x7f7dec14a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.215+0000 7f7df1a9a640 1 -- 172.21.15.94:0/2568730612 shutdown_connections 2024-03-29T10:44:37.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.215+0000 7f7df1a9a640 1 -- 172.21.15.94:0/2568730612 >> 172.21.15.94:0/2568730612 conn(0x7f7dec0b4930 msgr2=0x7f7dec0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:37.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.216+0000 7f7df1a9a640 1 -- 172.21.15.94:0/2568730612 shutdown_connections 2024-03-29T10:44:37.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.216+0000 7f7df1a9a640 1 -- 172.21.15.94:0/2568730612 wait complete. 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 10 monclient: build_initial_monmap 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 10 monclient: monmap: 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 1 librados: starting msgr at 2024-03-29T10:44:37.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 1 librados: starting objecter 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 1 Processor -- start 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 1 -- start start 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 1 librados: setting wanted keys 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 1 librados: calling monclient init 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.217+0000 7f7df1a9a640 10 monclient: init 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec06f050 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec06f430 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec070240 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec13b7a0 con 0x7f7dec06f050 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec0ab6b0 con 0x7f7dec06f430 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec0ab190 con 0x7f7dec070240 2024-03-29T10:44:37.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.218+0000 7f7df1a9a640 10 monclient(hunting): authenticate will time out at 1128.487889s 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7deb7fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f7dec06f050 0x7f7dec0ab8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:48546/0 (socket says 172.21.15.94:48546) 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7deb7fe640 1 -- 172.21.15.94:0/287828051 learned_addr learned my addr 172.21.15.94:0/287828051 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 client.?.objecter ms_handle_connect 0x7f7dec06f050 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396580021 0 0) 0x7f7dec13b7a0 con 0x7f7dec06f050 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd8004eb0 con 0x7f7dec06f050 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 client.?.objecter ms_handle_connect 0x7f7dec070240 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.219+0000 7f7dd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446757058 0 0) 0x7f7dec0ab190 con 0x7f7dec070240 2024-03-29T10:44:37.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd80051d0 con 0x7f7dec070240 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462896110 0 0) 0x7f7dd8004eb0 con 0x7f7dec06f050 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dec0ab190 con 0x7f7dec06f050 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7ddc001db0 con 0x7f7dec06f050 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): dump: 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:37.212 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4087611568 0 0) 0x7f7dec0ab190 con 0x7f7dec06f050 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 >> v1:172.21.15.94:6790/0 conn(0x7f7dec070240 legacy=0x7f7dec13b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 >> v1:172.21.15.173:6789/0 conn(0x7f7dec06f430 legacy=0x7f7dec13a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:37.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 monclient: found mon.a 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec13c4e0 con 0x7f7dec06f050 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 5 monclient: authenticate success, global_id 4428 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 10 monclient: _renew_subs 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec0aadb0 con 0x7f7dec06f050 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 10 monclient: _renew_subs 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:37.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec13bc20 con 0x7f7dec06f050 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7df1a9a640 1 librados: init done 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.220+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ddc002010 con 0x7f7dec06f050 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.221+0000 7f7dd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.221+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7ddc003dc0 con 0x7f7dec06f050 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.221+0000 7f7dd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.221+0000 7f7dd3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.221+0000 7f7dd3fff640 10 monclient: dump: 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:37.215 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.221+0000 7f7de97fa640 4 set_mon_vals no callback set 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.222+0000 7f7df1a9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.222+0000 7f7df1a9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.222+0000 7f7df1a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.222+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dcc005080 con 0x7f7dec06f050 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.223+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f7ddc003f20 con 0x7f7dec06f050 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.224+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f7ddc084e10 con 0x7f7dec06f050 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.224+0000 7f7dd3fff640 10 client.4428.objecter ms_dispatch 0x7f7dec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.224+0000 7f7dd3fff640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:37.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.224+0000 7f7dd3fff640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:37.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.224+0000 7f7deaffd640 10 monclient: get_auth_request con 0x7f7dd806dfd0 auth_method 0 2024-03-29T10:44:37.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.224+0000 7f7dd3fff640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T10:44:37.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.224+0000 7f7dd3fff640 10 client.4428.objecter ms_handle_connect 0x7f7dd806dfd0 2024-03-29T10:44:37.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.231+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ddc050520 con 0x7f7dec06f050 2024-03-29T10:44:37.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.231+0000 7f7dd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:37.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.231+0000 7f7dd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:37.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.548+0000 7f7df1a9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:37.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.548+0000 7f7df1a9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:37.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.548+0000 7f7df1a9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:37.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.548+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dcc005330 con 0x7f7dec06f050 2024-03-29T10:44:37.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.549+0000 7f7dd3fff640 1 -- 172.21.15.94:0/287828051 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f7ddc052620 con 0x7f7dec06f050 2024-03-29T10:44:37.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.549+0000 7f7dd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:37.534 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.549+0000 7f7dd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:37.534 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:37.534 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:37.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 10 librados: watch_flush enter 2024-03-29T10:44:37.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 10 librados: watch_flush exit 2024-03-29T10:44:37.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T10:44:37.536 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 10 client.4428.objecter successfully canceled tick 2024-03-29T10:44:37.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 >> v1:172.21.15.94:6816/3829934808 conn(0x7f7dd806dfd0 legacy=0x7f7dd8070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 10 monclient: shutdown 2024-03-29T10:44:37.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:37.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.552+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 >> v1:172.21.15.94:6789/0 conn(0x7f7dec06f050 legacy=0x7f7dec0ab8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:37.537 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.553+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 shutdown_connections 2024-03-29T10:44:37.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.553+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 >> 172.21.15.94:0/287828051 conn(0x7f7dec0b4930 msgr2=0x7f7dec06b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:37.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.553+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 shutdown_connections 2024-03-29T10:44:37.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.553+0000 7f7df1a9a640 1 -- 172.21.15.94:0/287828051 wait complete. 2024-03-29T10:44:37.538 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:37.553+0000 7f7df1a9a640 1 librados: shutdown 2024-03-29T10:44:37.553 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:44.360 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.515+0000 7f6e5648d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.515+0000 7f6e5648d640 10 monclient: build_initial_monmap 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.515+0000 7f6e5648d640 10 monclient: monmap: 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:44.516517+0000 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:44.516517+0000 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:44.499 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:44.500 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:44.500 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:44.500 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:44.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.517+0000 7f6e5648d640 1 Processor -- start 2024-03-29T10:44:44.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.517+0000 7f6e5648d640 1 -- start start 2024-03-29T10:44:44.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.517+0000 7f6e5648d640 10 monclient: init 2024-03-29T10:44:44.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:44.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:44.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e500a9930 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:44.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e50149dc0 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:44.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e5014ff40 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:44.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e500aae60 con 0x7f6e5014ff40 2024-03-29T10:44:44.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e500b4310 con 0x7f6e50149dc0 2024-03-29T10:44:44.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e500afdd0 con 0x7f6e500a9930 2024-03-29T10:44:44.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:44.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.518+0000 7f6e5648d640 10 monclient(hunting): authenticate will time out at 1135.786916s 2024-03-29T10:44:44.504 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e54a2a640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6e5014ff40 0x7f6e50152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:58865/0 (socket says 172.21.15.94:58865) 2024-03-29T10:44:44.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e54a2a640 1 -- 172.21.15.94:0/3474806108 learned_addr learned my addr 172.21.15.94:0/3474806108 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:44.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250947002 0 0) 0x7f6e500aae60 con 0x7f6e5014ff40 2024-03-29T10:44:44.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:44.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:44.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:44.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e4effd640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:44:44.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.522+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e3c003520 con 0x7f6e5014ff40 2024-03-29T10:44:44.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600310893 0 0) 0x7f6e3c003520 con 0x7f6e5014ff40 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 >> v1:172.21.15.94:6790/0 conn(0x7f6e500a9930 legacy=0x7f6e50143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 >> v1:172.21.15.173:6789/0 conn(0x7f6e50149dc0 legacy=0x7f6e5014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 1 monclient: found mon.noname-a 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5014a950 con 0x7f6e5014ff40 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 10 monclient: _finish_auth 0 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e38001690 con 0x7f6e5014ff40 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.523+0000 7f6e4effd640 10 monclient: dump: 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.524+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e38001860 con 0x7f6e5014ff40 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.524+0000 7f6e4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.524+0000 7f6e4effd640 1 -- 172.21.15.94:0/3474806108 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e38001c70 con 0x7f6e5014ff40 2024-03-29T10:44:44.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.524+0000 7f6e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.524+0000 7f6e4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.524+0000 7f6e4effd640 10 monclient: dump: 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:44.509 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:44.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.525+0000 7f6e5648d640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:44:44.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.525+0000 7f6e5648d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:44.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.525+0000 7f6e5648d640 4 set_mon_vals no callback set 2024-03-29T10:44:44.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.526+0000 7f6e5648d640 10 monclient: shutdown 2024-03-29T10:44:44.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.526+0000 7f6e5648d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:44.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.526+0000 7f6e5648d640 1 -- 172.21.15.94:0/3474806108 >> v1:172.21.15.94:6789/0 conn(0x7f6e5014ff40 legacy=0x7f6e50152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.526+0000 7f6e5648d640 1 -- 172.21.15.94:0/3474806108 shutdown_connections 2024-03-29T10:44:44.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.526+0000 7f6e5648d640 1 -- 172.21.15.94:0/3474806108 >> 172.21.15.94:0/3474806108 conn(0x7f6e500b35e0 msgr2=0x7f6e500b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:44.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.526+0000 7f6e5648d640 1 -- 172.21.15.94:0/3474806108 shutdown_connections 2024-03-29T10:44:44.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.526+0000 7f6e5648d640 1 -- 172.21.15.94:0/3474806108 wait complete. 2024-03-29T10:44:44.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.529+0000 7f6e5648d640 10 monclient: build_initial_monmap 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.529+0000 7f6e5648d640 10 monclient: monmap: 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.530+0000 7f6e5648d640 1 librados: starting msgr at 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.530+0000 7f6e5648d640 1 librados: starting objecter 2024-03-29T10:44:44.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.530+0000 7f6e5648d640 1 Processor -- start 2024-03-29T10:44:44.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.530+0000 7f6e5648d640 1 -- start start 2024-03-29T10:44:44.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.530+0000 7f6e5648d640 1 librados: setting wanted keys 2024-03-29T10:44:44.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.530+0000 7f6e5648d640 1 librados: calling monclient init 2024-03-29T10:44:44.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.530+0000 7f6e5648d640 10 monclient: init 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e500a9930 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e50149dc0 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5014ff40 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5013ba20 con 0x7f6e50149dc0 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5013ceb0 con 0x7f6e5014ff40 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5013d7e0 con 0x7f6e500a9930 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.531+0000 7f6e5648d640 10 monclient(hunting): authenticate will time out at 1135.799765s 2024-03-29T10:44:44.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f6e500a9930 0x7f6e5013c7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35176/0 (socket says 172.21.15.94:35176) 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4ffff640 1 -- 172.21.15.94:0/1574847597 learned_addr learned my addr 172.21.15.94:0/1574847597 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 10 client.?.objecter ms_handle_connect 0x7f6e500a9930 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180707596 0 0) 0x7f6e5013d7e0 con 0x7f6e500a9930 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.532+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e30004eb0 con 0x7f6e500a9930 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.533+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 151650277 0 0) 0x7f6e30004eb0 con 0x7f6e500a9930 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.533+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e5013d7e0 con 0x7f6e500a9930 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.533+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e40001a30 con 0x7f6e500a9930 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.533+0000 7f6e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.533+0000 7f6e4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:44.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.533+0000 7f6e4cff9640 10 monclient(hunting): dump: 2024-03-29T10:44:44.520 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:44.520 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:44.520 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.520 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.520 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.522 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:44.522 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:44.522 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:44.523 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:44.523 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:44.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.533+0000 7f6e4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:44.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.534+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1823132797 0 0) 0x7f6e5013d7e0 con 0x7f6e500a9930 2024-03-29T10:44:44.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.534+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 >> v1:172.21.15.173:6789/0 conn(0x7f6e5014ff40 legacy=0x7f6e500ac000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:44.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.535+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 >> v1:172.21.15.94:6789/0 conn(0x7f6e50149dc0 legacy=0x7f6e500ab8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.536+0000 7f6e4cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.536+0000 7f6e4cff9640 1 monclient: found mon.c 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.536+0000 7f6e4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e4cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5013bc70 con 0x7f6e500a9930 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e4cff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e4cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:44:44.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 10 monclient: _renew_subs 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5013dac0 con 0x7f6e500a9930 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 10 monclient: _renew_subs 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e501ef990 con 0x7f6e500a9930 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.537+0000 7f6e5648d640 1 librados: init done 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.538+0000 7f6e5648d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e5648d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e5648d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e500ab610 con 0x7f6e500a9930 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e40001d30 con 0x7f6e500a9930 2024-03-29T10:44:44.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6e40003a60 con 0x7f6e500a9930 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e4cff9640 10 monclient: dump: 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:44.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.539+0000 7f6e4e7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.540+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f6e40003c80 con 0x7f6e500a9930 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.541+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f6e40084b60 con 0x7f6e500a9930 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.541+0000 7f6e4cff9640 10 client.4451.objecter ms_dispatch 0x7f6e50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.541+0000 7f6e4cff9640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.541+0000 7f6e4cff9640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.541+0000 7f6e4cff9640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.541+0000 7f6e4f7fe640 10 monclient: get_auth_request con 0x7f6e3006dc50 auth_method 0 2024-03-29T10:44:44.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.542+0000 7f6e4cff9640 10 client.4451.objecter ms_handle_connect 0x7f6e3006dc50 2024-03-29T10:44:44.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.549+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e500ab610 con 0x7f6e500a9930 2024-03-29T10:44:44.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.549+0000 7f6e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:44.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.549+0000 7f6e4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:44.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.874+0000 7f6e5648d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:44.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.874+0000 7f6e5648d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:44.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.874+0000 7f6e5648d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:44.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.875+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e500ab610 con 0x7f6e500a9930 2024-03-29T10:44:44.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.875+0000 7f6e4cff9640 1 -- 172.21.15.94:0/1574847597 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f6e500ab610 con 0x7f6e500a9930 2024-03-29T10:44:44.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.875+0000 7f6e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:44.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.875+0000 7f6e4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:44.860 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:44.860 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:44.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.879+0000 7f6e5648d640 10 librados: watch_flush enter 2024-03-29T10:44:44.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.880+0000 7f6e5648d640 10 librados: watch_flush exit 2024-03-29T10:44:44.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.880+0000 7f6e5648d640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T10:44:44.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.880+0000 7f6e5648d640 10 client.4451.objecter successfully canceled tick 2024-03-29T10:44:44.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 >> v1:172.21.15.94:6816/3829934808 conn(0x7f6e3006dc50 legacy=0x7f6e30070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 10 monclient: shutdown 2024-03-29T10:44:44.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:44.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 >> v1:172.21.15.94:6790/0 conn(0x7f6e500a9930 legacy=0x7f6e5013c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 shutdown_connections 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 >> 172.21.15.94:0/1574847597 conn(0x7f6e500b35e0 msgr2=0x7f6e500b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 shutdown_connections 2024-03-29T10:44:44.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.881+0000 7f6e5648d640 1 -- 172.21.15.94:0/1574847597 wait complete. 2024-03-29T10:44:44.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:44.882+0000 7f6e5648d640 1 librados: shutdown 2024-03-29T10:44:44.883 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:51.690 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.858+0000 7fc18e59f640 10 monclient: get_monmap_and_config 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.858+0000 7fc18e59f640 10 monclient: build_initial_monmap 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.858+0000 7fc18e59f640 10 monclient: monmap: 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:51.859005+0000 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:51.859005+0000 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:51.841 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:51.842 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:51.842 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.859+0000 7fc18e59f640 1 Processor -- start 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.860+0000 7fc18e59f640 1 -- start start 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.860+0000 7fc18e59f640 10 monclient: init 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 10 monclient(hunting): picked mon.noname-a con 0x7fc1800389e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 10 monclient(hunting): picked mon.noname-c con 0x7fc180038dc0 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 10 monclient(hunting): picked mon.noname-b con 0x7fc180030660 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1800c54c0 con 0x7fc1800389e0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc180036430 con 0x7fc180038dc0 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc180039840 con 0x7fc180030660 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:51.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.861+0000 7fc18e59f640 10 monclient(hunting): authenticate will time out at 1143.129235s 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc18d59d640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc1800389e0 0x7fc180033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36927/0 (socket says 172.21.15.94:36927) 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc18d59d640 1 -- 172.21.15.94:0/685203843 learned_addr learned my addr 172.21.15.94:0/685203843 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378649320 0 0) 0x7fc1800c54c0 con 0x7fc1800389e0 2024-03-29T10:44:51.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc17ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:51.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc17ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:51.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc17ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:51.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc17ffff640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:44:51.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.863+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc174003520 con 0x7fc1800389e0 2024-03-29T10:44:51.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.864+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1958162196 0 0) 0x7fc174003520 con 0x7fc1800389e0 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.864+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 >> v1:172.21.15.94:6790/0 conn(0x7fc180030660 legacy=0x7fc1800d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.864+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 >> v1:172.21.15.173:6789/0 conn(0x7fc180038dc0 legacy=0x7fc1800cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 1 monclient: found mon.noname-a 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc180035690 con 0x7fc1800389e0 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 10 monclient: _finish_auth 0 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc18804f730 con 0x7fc1800389e0 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc17ffff640 10 monclient: dump: 2024-03-29T10:44:51.853 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc18e59f640 5 monclient: authenticate success, global_id 4434 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.867+0000 7fc18e59f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1880008d0 con 0x7fc1800389e0 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc17ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:51.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc17ffff640 1 -- 172.21.15.94:0/685203843 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc1880758b0 con 0x7fc1800389e0 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc17ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc17ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc17ffff640 10 monclient: dump: 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:51.855 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc18e59f640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.868+0000 7fc18e59f640 4 set_mon_vals no callback set 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.869+0000 7fc18e59f640 10 monclient: shutdown 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.869+0000 7fc18e59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.869+0000 7fc18e59f640 1 -- 172.21.15.94:0/685203843 >> v1:172.21.15.94:6789/0 conn(0x7fc1800389e0 legacy=0x7fc180033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.869+0000 7fc18e59f640 1 -- 172.21.15.94:0/685203843 shutdown_connections 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.869+0000 7fc18e59f640 1 -- 172.21.15.94:0/685203843 >> 172.21.15.94:0/685203843 conn(0x7fc180005c00 msgr2=0x7fc180006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.869+0000 7fc18e59f640 1 -- 172.21.15.94:0/685203843 shutdown_connections 2024-03-29T10:44:51.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.869+0000 7fc18e59f640 1 -- 172.21.15.94:0/685203843 wait complete. 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.870+0000 7fc18e59f640 10 monclient: build_initial_monmap 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.870+0000 7fc18e59f640 10 monclient: monmap: 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:51.857 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.871+0000 7fc18e59f640 1 librados: starting msgr at 2024-03-29T10:44:51.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.871+0000 7fc18e59f640 1 librados: starting objecter 2024-03-29T10:44:51.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.871+0000 7fc18e59f640 1 Processor -- start 2024-03-29T10:44:51.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.871+0000 7fc18e59f640 1 -- start start 2024-03-29T10:44:51.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.871+0000 7fc18e59f640 1 librados: setting wanted keys 2024-03-29T10:44:51.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.871+0000 7fc18e59f640 1 librados: calling monclient init 2024-03-29T10:44:51.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.871+0000 7fc18e59f640 10 monclient: init 2024-03-29T10:44:51.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 10 monclient(hunting): picked mon.noname-c con 0x7fc180030660 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 10 monclient(hunting): picked mon.noname-a con 0x7fc1800389e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 10 monclient(hunting): picked mon.noname-b con 0x7fc1800c79a0 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc180009e80 con 0x7fc1800389e0 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1800c83a0 con 0x7fc1800c79a0 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1800c28c0 con 0x7fc180030660 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18e59f640 10 monclient(hunting): authenticate will time out at 1143.140591s 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18d59d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc180030660 0x7fc180160f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:59924/0 (socket says 172.21.15.94:59924) 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc18d59d640 1 -- 172.21.15.94:0/1160410646 learned_addr learned my addr 172.21.15.94:0/1160410646 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc17dffb640 10 client.?.objecter ms_handle_connect 0x7fc180030660 2024-03-29T10:44:51.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.872+0000 7fc17dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.873+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579618530 0 0) 0x7fc1800c28c0 con 0x7fc180030660 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.873+0000 7fc17dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.873+0000 7fc17dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.873+0000 7fc17dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:51.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.873+0000 7fc17dffb640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.873+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc16c004eb0 con 0x7fc180030660 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304396613 0 0) 0x7fc16c004eb0 con 0x7fc180030660 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1800c28c0 con 0x7fc180030660 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc18804f400 con 0x7fc180030660 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 10 monclient(hunting): dump: 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:51.865 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1660266384 0 0) 0x7fc1800c28c0 con 0x7fc180030660 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 >> v1:172.21.15.173:6789/0 conn(0x7fc1800c79a0 legacy=0x7fc1800c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.876+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 >> v1:172.21.15.94:6789/0 conn(0x7fc1800389e0 legacy=0x7fc1800c1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 1 monclient: found mon.c 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18000a0f0 con 0x7fc180030660 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:51.866 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc17dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 10 monclient: _renew_subs 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1800c7e80 con 0x7fc180030660 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 10 monclient: _renew_subs 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1800c7040 con 0x7fc180030660 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.877+0000 7fc18e59f640 1 librados: init done 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc18804fdd0 con 0x7fc180030660 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc17dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc188067bf0 con 0x7fc180030660 2024-03-29T10:44:51.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc17dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc17dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc17dffb640 10 monclient: dump: 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc17effd640 4 set_mon_vals no callback set 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.878+0000 7fc18e59f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:51.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.879+0000 7fc18e59f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.879+0000 7fc18e59f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.879+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc168005080 con 0x7fc180030660 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.880+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fc188067d50 con 0x7fc180030660 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.880+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fc1880ff700 con 0x7fc180030660 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.880+0000 7fc17dffb640 10 client.4454.objecter ms_dispatch 0x7fc188000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.880+0000 7fc17dffb640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.880+0000 7fc17dffb640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.880+0000 7fc17dffb640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.880+0000 7fc18cd9c640 10 monclient: get_auth_request con 0x7fc16c06dc00 auth_method 0 2024-03-29T10:44:51.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.881+0000 7fc17dffb640 10 client.4454.objecter ms_handle_connect 0x7fc16c06dc00 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.889+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1880cafe0 con 0x7fc180030660 2024-03-29T10:44:51.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.889+0000 7fc17dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:51.872 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:51.889+0000 7fc17dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:52.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.214+0000 7fc18e59f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:52.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.214+0000 7fc18e59f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:52.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.214+0000 7fc18e59f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:44:52.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.214+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc168005330 con 0x7fc180030660 2024-03-29T10:44:52.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.215+0000 7fc17dffb640 1 -- 172.21.15.94:0/1160410646 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fc1880ccf10 con 0x7fc180030660 2024-03-29T10:44:52.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.215+0000 7fc17dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:52.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.215+0000 7fc17dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:52.200 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:52.200 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:52.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.221+0000 7fc18e59f640 10 librados: watch_flush enter 2024-03-29T10:44:52.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.221+0000 7fc18e59f640 10 librados: watch_flush exit 2024-03-29T10:44:52.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.221+0000 7fc18e59f640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T10:44:52.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.221+0000 7fc18e59f640 10 client.4454.objecter successfully canceled tick 2024-03-29T10:44:52.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.222+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 >> v1:172.21.15.94:6816/3829934808 conn(0x7fc16c06dc00 legacy=0x7fc16c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:52.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.222+0000 7fc18e59f640 10 monclient: shutdown 2024-03-29T10:44:52.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.222+0000 7fc18e59f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:52.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.222+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 >> v1:172.21.15.94:6790/0 conn(0x7fc180030660 legacy=0x7fc180160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:52.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.225+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 shutdown_connections 2024-03-29T10:44:52.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.225+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 >> 172.21.15.94:0/1160410646 conn(0x7fc180005c00 msgr2=0x7fc1800c3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:52.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.232+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 shutdown_connections 2024-03-29T10:44:52.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.232+0000 7fc18e59f640 1 -- 172.21.15.94:0/1160410646 wait complete. 2024-03-29T10:44:52.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:52.232+0000 7fc18e59f640 1 librados: shutdown 2024-03-29T10:44:52.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:59.039 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:59.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.213+0000 7fc05666a640 10 monclient: get_monmap_and_config 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.213+0000 7fc05666a640 10 monclient: build_initial_monmap 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.213+0000 7fc05666a640 10 monclient: monmap: 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:44:59.214828+0000 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:44:59.214828+0000 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:44:59.197 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:59.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.215+0000 7fc05666a640 1 Processor -- start 2024-03-29T10:44:59.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.215+0000 7fc05666a640 1 -- start start 2024-03-29T10:44:59.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.215+0000 7fc05666a640 10 monclient: init 2024-03-29T10:44:59.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.216+0000 7fc05666a640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:59.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.216+0000 7fc05666a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:59.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.216+0000 7fc05666a640 10 monclient(hunting): picked mon.noname-b con 0x7fc0501457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:59.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.219+0000 7fc05666a640 10 monclient(hunting): picked mon.noname-c con 0x7fc05013f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc05666a640 10 monclient(hunting): picked mon.noname-a con 0x7fc0500a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc054c07640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc0500a9d70 0x7fc05014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42281/0 (socket says 172.21.15.94:42281) 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc04ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc0501457f0 0x7fc05013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:45677/0 (socket says 172.21.15.94:45677) 2024-03-29T10:44:59.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc054c07640 1 -- 172.21.15.94:0/1092585059 learned_addr learned my addr 172.21.15.94:0/1092585059 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:59.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0500aaef0 con 0x7fc0500a9d70 2024-03-29T10:44:59.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050145bd0 con 0x7fc05013f650 2024-03-29T10:44:59.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050145eb0 con 0x7fc0501457f0 2024-03-29T10:44:59.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc05666a640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:59.206 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.222+0000 7fc05666a640 10 monclient(hunting): authenticate will time out at 1150.490714s 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028557501 0 0) 0x7fc0500aaef0 con 0x7fc0500a9d70 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.208 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc038003520 con 0x7fc0500a9d70 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812253786 0 0) 0x7fc050145eb0 con 0x7fc0501457f0 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0500aaef0 con 0x7fc0501457f0 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402273268 0 0) 0x7fc050145bd0 con 0x7fc05013f650 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:44:59.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.225+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050145eb0 con 0x7fc05013f650 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681928581 0 0) 0x7fc0500aaef0 con 0x7fc0501457f0 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 >> v1:172.21.15.173:6789/0 conn(0x7fc05013f650 legacy=0x7fc0500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 >> v1:172.21.15.94:6789/0 conn(0x7fc0500a9d70 legacy=0x7fc05014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 1 monclient: found mon.noname-b 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc050146340 con 0x7fc0501457f0 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 10 monclient: _finish_auth 0 2024-03-29T10:44:59.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc040001520 con 0x7fc0501457f0 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.226+0000 7fc04effd640 10 monclient: dump: 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:59.211 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:59.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.229+0000 7fc05666a640 5 monclient: authenticate success, global_id 4457 2024-03-29T10:44:59.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.229+0000 7fc05666a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.229+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc040001680 con 0x7fc0501457f0 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.229+0000 7fc04effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.229+0000 7fc04effd640 1 -- 172.21.15.94:0/1092585059 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc040001a90 con 0x7fc0501457f0 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.229+0000 7fc05666a640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.229+0000 7fc05666a640 4 set_mon_vals no callback set 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.230+0000 7fc05666a640 10 monclient: shutdown 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.230+0000 7fc04effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.230+0000 7fc04effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.230+0000 7fc04effd640 10 monclient: dump: 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.230+0000 7fc05666a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:59.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.230+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 >> v1:172.21.15.94:6790/0 conn(0x7fc0501457f0 legacy=0x7fc05013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.231+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 shutdown_connections 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.231+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 >> 172.21.15.94:0/1092585059 conn(0x7fc050066a00 msgr2=0x7fc050066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.231+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 shutdown_connections 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.231+0000 7fc05666a640 1 -- 172.21.15.94:0/1092585059 wait complete. 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.232+0000 7fc05666a640 10 monclient: build_initial_monmap 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.232+0000 7fc05666a640 10 monclient: monmap: 2024-03-29T10:44:59.215 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:44:59.216 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.233+0000 7fc05666a640 1 librados: starting msgr at 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.233+0000 7fc05666a640 1 librados: starting objecter 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.233+0000 7fc05666a640 1 Processor -- start 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.233+0000 7fc05666a640 1 -- start start 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.234+0000 7fc05666a640 1 librados: setting wanted keys 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.234+0000 7fc05666a640 1 librados: calling monclient init 2024-03-29T10:44:59.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.234+0000 7fc05666a640 10 monclient: init 2024-03-29T10:44:59.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.234+0000 7fc05666a640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:59.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.234+0000 7fc05666a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:59.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.238+0000 7fc05666a640 10 monclient(hunting): picked mon.noname-c con 0x7fc0500a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.238+0000 7fc05666a640 10 monclient(hunting): picked mon.noname-a con 0x7fc05013f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc05666a640 10 monclient(hunting): picked mon.noname-b con 0x7fc0501457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc05666a640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050142970 con 0x7fc05013f650 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc05666a640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050141c50 con 0x7fc0501457f0 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc05666a640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050141de0 con 0x7fc0500a9d70 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc05666a640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc04ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc0500a9d70 0x7fc050142080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:45771/0 (socket says 172.21.15.94:45771) 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc05666a640 10 monclient(hunting): authenticate will time out at 1150.508816s 2024-03-29T10:44:59.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.240+0000 7fc04ffff640 1 -- 172.21.15.94:0/3813005548 learned_addr learned my addr 172.21.15.94:0/3813005548 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.244+0000 7fc04cff9640 10 client.?.objecter ms_handle_connect 0x7fc0500a9d70 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.244+0000 7fc04cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.244+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417454760 0 0) 0x7fc050141de0 con 0x7fc0500a9d70 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.244+0000 7fc04cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.244+0000 7fc04cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c004eb0 con 0x7fc0500a9d70 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 client.?.objecter ms_handle_connect 0x7fc05013f650 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:59.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534804165 0 0) 0x7fc050142970 con 0x7fc05013f650 2024-03-29T10:44:59.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:59.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:59.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:59.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:44:59.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050141de0 con 0x7fc05013f650 2024-03-29T10:44:59.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3137182890 0 0) 0x7fc03c004eb0 con 0x7fc0500a9d70 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.245+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc050142970 con 0x7fc0500a9d70 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc040001380 con 0x7fc0500a9d70 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 10 monclient(hunting): dump: 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:59.237 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027934576 0 0) 0x7fc050141de0 con 0x7fc05013f650 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc03c004eb0 con 0x7fc05013f650 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc044001760 con 0x7fc05013f650 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.246+0000 7fc04cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:59.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 10 monclient(hunting): dump: 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 10 client.?.objecter ms_handle_connect 0x7fc0501457f0 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1259064858 0 0) 0x7fc03c004eb0 con 0x7fc05013f650 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 >> v1:172.21.15.94:6790/0 conn(0x7fc0500a9d70 legacy=0x7fc050142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 >> v1:172.21.15.173:6789/0 conn(0x7fc0501457f0 legacy=0x7fc0501da470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.250+0000 7fc04cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 1 monclient: found mon.a 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0501374d0 con 0x7fc05013f650 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc044001c70 con 0x7fc05013f650 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc0440037b0 con 0x7fc05013f650 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.251+0000 7fc04cff9640 10 monclient: dump: 2024-03-29T10:44:59.240 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.252+0000 7fc04e7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.252+0000 7fc05666a640 5 monclient: authenticate success, global_id 4440 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 10 monclient: _renew_subs 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc050137720 con 0x7fc05013f650 2024-03-29T10:44:59.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:59.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 10 monclient: _renew_subs 2024-03-29T10:44:59.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:59.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc050144940 con 0x7fc05013f650 2024-03-29T10:44:59.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.253+0000 7fc05666a640 1 librados: init done 2024-03-29T10:44:59.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.256+0000 7fc05666a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.256+0000 7fc05666a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.256+0000 7fc05666a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.256+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc050075f10 con 0x7fc05013f650 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.259+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fc044003910 con 0x7fc05013f650 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.259+0000 7fc04ffff640 10 monclient: get_auth_request con 0x7fc03c06ded0 auth_method 0 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.260+0000 7fc04cff9640 10 client.4440.objecter ms_handle_connect 0x7fc03c06ded0 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.260+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fc0440836d0 con 0x7fc05013f650 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.260+0000 7fc04cff9640 10 client.4440.objecter ms_dispatch 0x7fc050000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.260+0000 7fc04cff9640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.260+0000 7fc04cff9640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:59.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.260+0000 7fc04cff9640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T10:44:59.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.267+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc04404efb0 con 0x7fc05013f650 2024-03-29T10:44:59.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.267+0000 7fc04cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:59.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.267+0000 7fc04cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:59.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.587+0000 7fc05666a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:59.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.587+0000 7fc05666a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:59.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.587+0000 7fc05666a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:44:59.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.587+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0500b3680 con 0x7fc05013f650 2024-03-29T10:44:59.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.588+0000 7fc04cff9640 1 -- 172.21.15.94:0/3813005548 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fc044050ee0 con 0x7fc05013f650 2024-03-29T10:44:59.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.588+0000 7fc04cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:59.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.588+0000 7fc04cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:59.573 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:44:59.573 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:59.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 10 librados: watch_flush enter 2024-03-29T10:44:59.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 10 librados: watch_flush exit 2024-03-29T10:44:59.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T10:44:59.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 10 client.4440.objecter successfully canceled tick 2024-03-29T10:44:59.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 >> v1:172.21.15.94:6816/3829934808 conn(0x7fc03c06ded0 legacy=0x7fc03c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 10 monclient: shutdown 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.594+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 >> v1:172.21.15.94:6789/0 conn(0x7fc05013f650 legacy=0x7fc050136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.595+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 shutdown_connections 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.595+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 >> 172.21.15.94:0/3813005548 conn(0x7fc050066a00 msgr2=0x7fc05013c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.595+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 shutdown_connections 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.595+0000 7fc05666a640 1 -- 172.21.15.94:0/3813005548 wait complete. 2024-03-29T10:44:59.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:44:59.595+0000 7fc05666a640 1 librados: shutdown 2024-03-29T10:44:59.597 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:06.404 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.570+0000 7f0dcdda8640 10 monclient: get_monmap_and_config 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.570+0000 7f0dcdda8640 10 monclient: build_initial_monmap 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.570+0000 7f0dcdda8640 10 monclient: monmap: 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:06.570952+0000 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:06.570952+0000 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:06.554 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:06.555 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:06.555 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:06.555 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:06.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.572+0000 7f0dcdda8640 1 Processor -- start 2024-03-29T10:45:06.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.572+0000 7f0dcdda8640 1 -- start start 2024-03-29T10:45:06.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.572+0000 7f0dcdda8640 10 monclient: init 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.573+0000 7f0dcdda8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.573+0000 7f0dcdda8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.573+0000 7f0dcdda8640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc80a9ac0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.574+0000 7f0dcdda8640 10 monclient(hunting): picked mon.noname-c con 0x7f0dc813ee90 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.574+0000 7f0dcdda8640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc80af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:06.556 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.574+0000 7f0dcdda8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc80afd10 con 0x7f0dc80af900 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.574+0000 7f0dcdda8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc80af6f0 con 0x7f0dc813ee90 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.574+0000 7f0dcdda8640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc80aac80 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.574+0000 7f0dcdda8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:06.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.574+0000 7f0dcdda8640 10 monclient(hunting): authenticate will time out at 1157.842283s 2024-03-29T10:45:06.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.577+0000 7f0dc6ffd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0dc813ee90 0x7f0dc813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:54035/0 (socket says 172.21.15.94:54035) 2024-03-29T10:45:06.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.577+0000 7f0dc6ffd640 1 -- 172.21.15.94:0/2622469225 learned_addr learned my addr 172.21.15.94:0/2622469225 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232511931 0 0) 0x7f0dc80af6f0 con 0x7f0dc813ee90 2024-03-29T10:45:06.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:06.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:06.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:06.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:45:06.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db0003520 con 0x7f0dc813ee90 2024-03-29T10:45:06.562 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2695933279 0 0) 0x7f0db0003520 con 0x7f0dc813ee90 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 >> v1:172.21.15.94:6790/0 conn(0x7f0dc80a9ac0 legacy=0x7f0dc80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 >> v1:172.21.15.94:6789/0 conn(0x7f0dc80af900 legacy=0x7f0dc8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 monclient: found mon.noname-c 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc80aaf60 con 0x7f0dc813ee90 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0dbc001690 con 0x7f0dc813ee90 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.578+0000 7f0dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 10 monclient: dump: 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dcdda8640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dcdda8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dbc0019a0 con 0x7f0dc813ee90 2024-03-29T10:45:06.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 1 -- 172.21.15.94:0/2622469225 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0dbc001d30 con 0x7f0dc813ee90 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dcdda8640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dcdda8640 4 set_mon_vals no callback set 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.579+0000 7f0dc67fc640 10 monclient: dump: 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:06.565 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:06.566 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:06.566 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:06.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.580+0000 7f0dcdda8640 10 monclient: shutdown 2024-03-29T10:45:06.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.580+0000 7f0dcdda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:06.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.580+0000 7f0dcdda8640 1 -- 172.21.15.94:0/2622469225 >> v1:172.21.15.173:6789/0 conn(0x7f0dc813ee90 legacy=0x7f0dc813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.580+0000 7f0dcdda8640 1 -- 172.21.15.94:0/2622469225 shutdown_connections 2024-03-29T10:45:06.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.580+0000 7f0dcdda8640 1 -- 172.21.15.94:0/2622469225 >> 172.21.15.94:0/2622469225 conn(0x7f0dc80b35e0 msgr2=0x7f0dc80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:06.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.584+0000 7f0dcdda8640 1 -- 172.21.15.94:0/2622469225 shutdown_connections 2024-03-29T10:45:06.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.584+0000 7f0dcdda8640 1 -- 172.21.15.94:0/2622469225 wait complete. 2024-03-29T10:45:06.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.585+0000 7f0dcdda8640 10 monclient: build_initial_monmap 2024-03-29T10:45:06.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.585+0000 7f0dcdda8640 10 monclient: monmap: 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:06.568 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:06.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.586+0000 7f0dcdda8640 1 librados: starting msgr at 2024-03-29T10:45:06.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.586+0000 7f0dcdda8640 1 librados: starting objecter 2024-03-29T10:45:06.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.586+0000 7f0dcdda8640 1 Processor -- start 2024-03-29T10:45:06.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.586+0000 7f0dcdda8640 1 -- start start 2024-03-29T10:45:06.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.586+0000 7f0dcdda8640 1 librados: setting wanted keys 2024-03-29T10:45:06.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.586+0000 7f0dcdda8640 1 librados: calling monclient init 2024-03-29T10:45:06.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.586+0000 7f0dcdda8640 10 monclient: init 2024-03-29T10:45:06.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.587+0000 7f0dcdda8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:06.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.587+0000 7f0dcdda8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:06.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.587+0000 7f0dcdda8640 10 monclient(hunting): picked mon.noname-a con 0x7f0dc80a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dc77fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f0dc80a9ac0 0x7f0dc813c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:59551/0 (socket says 172.21.15.94:59551) 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dcdda8640 10 monclient(hunting): picked mon.noname-b con 0x7f0dc80af900 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dc77fe640 1 -- 172.21.15.94:0/1295750207 learned_addr learned my addr 172.21.15.94:0/1295750207 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dcdda8640 10 monclient(hunting): picked mon.noname-c con 0x7f0dc813ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc8141c20 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc8141e70 con 0x7f0dc80af900 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc813b8d0 con 0x7f0dc813ee90 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dcdda8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0dcdda8640 10 monclient(hunting): authenticate will time out at 1157.859990s 2024-03-29T10:45:06.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0daffff640 10 client.?.objecter ms_handle_connect 0x7f0dc80a9ac0 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.591+0000 7f0daffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670732346 0 0) 0x7f0dc8141c20 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4004eb0 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655488301 0 0) 0x7f0db4004eb0 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0dc8141c20 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0dc0001a60 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): dump: 2024-03-29T10:45:06.576 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.592+0000 7f0daffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:06.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.594+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 689426372 0 0) 0x7f0dc8141c20 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.594+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 >> v1:172.21.15.94:6790/0 conn(0x7f0dc813ee90 legacy=0x7f0dc814b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 >> v1:172.21.15.173:6789/0 conn(0x7f0dc80af900 legacy=0x7f0dc812e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 1 monclient: found mon.a 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc813ba60 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 10 monclient: _finish_auth 0 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0daffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0dcdda8640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0dcdda8640 10 monclient: _renew_subs 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0dcdda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:06.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc813c7d0 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0dcdda8640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:06.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0dcdda8640 10 monclient: _renew_subs 2024-03-29T10:45:06.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.596+0000 7f0dcdda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:06.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.597+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc8140dd0 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.597+0000 7f0dcdda8640 1 librados: init done 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.598+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0dc0002070 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.598+0000 7f0daffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.598+0000 7f0dc57fa640 4 set_mon_vals no callback set 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.598+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0dc0001a60 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.598+0000 7f0daffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.598+0000 7f0daffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.598+0000 7f0daffff640 10 monclient: dump: 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.582 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:06.583 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:06.583 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:06.583 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:06.583 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:06.583 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:06.583 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:06.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.600+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f0dc0004c30 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.600+0000 7f0dcdda8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:06.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.600+0000 7f0dcdda8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:06.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.600+0000 7f0dcdda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:06.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.600+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da8005080 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.600+0000 7f0dc6ffd640 10 monclient: get_auth_request con 0x7f0db40722d0 auth_method 0 2024-03-29T10:45:06.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.601+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0dc0054820 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.601+0000 7f0daffff640 10 client.4443.objecter ms_dispatch 0x7f0dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:06.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.601+0000 7f0daffff640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:06.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.601+0000 7f0daffff640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:06.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.601+0000 7f0daffff640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T10:45:06.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.601+0000 7f0daffff640 10 client.4443.objecter ms_handle_connect 0x7f0db40722d0 2024-03-29T10:45:06.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.609+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0dc0050ac0 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.609+0000 7f0daffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:06.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.609+0000 7f0daffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:06.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.930+0000 7f0dcdda8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:06.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.930+0000 7f0dcdda8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:06.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.930+0000 7f0dcdda8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:06.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.930+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0da8005330 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.933+0000 7f0daffff640 1 -- 172.21.15.94:0/1295750207 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f0dc0050c60 con 0x7f0dc80a9ac0 2024-03-29T10:45:06.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.933+0000 7f0daffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:06.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.933+0000 7f0daffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:06.916 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:06.917 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:06.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.937+0000 7f0dcdda8640 10 librados: watch_flush enter 2024-03-29T10:45:06.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.937+0000 7f0dcdda8640 10 librados: watch_flush exit 2024-03-29T10:45:06.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.937+0000 7f0dcdda8640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T10:45:06.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.937+0000 7f0dcdda8640 10 client.4443.objecter successfully canceled tick 2024-03-29T10:45:06.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.938+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 >> v1:172.21.15.94:6816/3829934808 conn(0x7f0db40722d0 legacy=0x7f0db4074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.938+0000 7f0dcdda8640 10 monclient: shutdown 2024-03-29T10:45:06.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.938+0000 7f0dcdda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:06.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.938+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 >> v1:172.21.15.94:6789/0 conn(0x7f0dc80a9ac0 legacy=0x7f0dc813c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:06.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.938+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 shutdown_connections 2024-03-29T10:45:06.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.938+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 >> 172.21.15.94:0/1295750207 conn(0x7f0dc80b35e0 msgr2=0x7f0dc80b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:06.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.941+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 shutdown_connections 2024-03-29T10:45:06.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.942+0000 7f0dcdda8640 1 -- 172.21.15.94:0/1295750207 wait complete. 2024-03-29T10:45:06.925 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:06.942+0000 7f0dcdda8640 1 librados: shutdown 2024-03-29T10:45:06.942 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:13.749 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.917+0000 7f3be9eba640 10 monclient: get_monmap_and_config 2024-03-29T10:45:13.899 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.917+0000 7f3be9eba640 10 monclient: build_initial_monmap 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.917+0000 7f3be9eba640 10 monclient: monmap: 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:13.917990+0000 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:13.917990+0000 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:13.900 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 1 Processor -- start 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 1 -- start start 2024-03-29T10:45:13.901 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient: init 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient(hunting): picked mon.noname-c con 0x7f3be40b2800 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:13.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient(hunting): picked mon.noname-b con 0x7f3be40a9c00 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient(hunting): picked mon.noname-a con 0x7f3be40b38c0 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be4141eb0 con 0x7f3be40b38c0 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be40b22d0 con 0x7f3be40b2800 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be4142d30 con 0x7f3be40a9c00 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.919+0000 7f3be9eba640 10 monclient(hunting): authenticate will time out at 1165.188086s 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.923+0000 7f3be3fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f3be40a9c00 0x7f3be413bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:45281/0 (socket says 172.21.15.94:45281) 2024-03-29T10:45:13.906 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.923+0000 7f3be3fff640 1 -- 172.21.15.94:0/734945992 learned_addr learned my addr 172.21.15.94:0/734945992 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:13.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010029353 0 0) 0x7f3be4142d30 con 0x7f3be40a9c00 2024-03-29T10:45:13.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:45:13.908 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bcc003520 con 0x7f3be40a9c00 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855267575 0 0) 0x7f3be4141eb0 con 0x7f3be40b38c0 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be4142d30 con 0x7f3be40b38c0 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154939788 0 0) 0x7f3be40b22d0 con 0x7f3be40b2800 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:45:13.909 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be4141eb0 con 0x7f3be40b2800 2024-03-29T10:45:13.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 321555456 0 0) 0x7f3be4142d30 con 0x7f3be40b38c0 2024-03-29T10:45:13.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 >> v1:172.21.15.94:6790/0 conn(0x7f3be40a9c00 legacy=0x7f3be413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 >> v1:172.21.15.173:6789/0 conn(0x7f3be40b2800 legacy=0x7f3be4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:13.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 monclient: found mon.noname-a 2024-03-29T10:45:13.910 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be40aaf90 con 0x7f3be40b38c0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3bd4001520 con 0x7f3be40b38c0 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: dump: 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.911 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be9eba640 5 monclient: authenticate success, global_id 4446 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be9eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd4001680 con 0x7f3be40b38c0 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be37fe640 1 -- 172.21.15.94:0/734945992 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3bd4001a10 con 0x7f3be40b38c0 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be9eba640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.926+0000 7f3be9eba640 4 set_mon_vals no callback set 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.927+0000 7f3be37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.927+0000 7f3be37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:13.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.927+0000 7f3be37fe640 10 monclient: dump: 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:13.913 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:13.914 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.927+0000 7f3be9eba640 10 monclient: shutdown 2024-03-29T10:45:13.914 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.927+0000 7f3be9eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:13.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.927+0000 7f3be9eba640 1 -- 172.21.15.94:0/734945992 >> v1:172.21.15.94:6789/0 conn(0x7f3be40b38c0 legacy=0x7f3be414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.931+0000 7f3be9eba640 1 -- 172.21.15.94:0/734945992 shutdown_connections 2024-03-29T10:45:13.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.931+0000 7f3be9eba640 1 -- 172.21.15.94:0/734945992 >> 172.21.15.94:0/734945992 conn(0x7f3be4066a00 msgr2=0x7f3be4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:13.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.931+0000 7f3be9eba640 1 -- 172.21.15.94:0/734945992 shutdown_connections 2024-03-29T10:45:13.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.931+0000 7f3be9eba640 1 -- 172.21.15.94:0/734945992 wait complete. 2024-03-29T10:45:13.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 10 monclient: build_initial_monmap 2024-03-29T10:45:13.915 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 10 monclient: monmap: 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 1 librados: starting msgr at 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 1 librados: starting objecter 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 1 Processor -- start 2024-03-29T10:45:13.916 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 1 -- start start 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 1 librados: setting wanted keys 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 1 librados: calling monclient init 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.932+0000 7f3be9eba640 10 monclient: init 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.934+0000 7f3be9eba640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.934+0000 7f3be9eba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:13.917 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.934+0000 7f3be9eba640 10 monclient(hunting): picked mon.noname-a con 0x7f3be40a9c00 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.936+0000 7f3be9eba640 10 monclient(hunting): picked mon.noname-b con 0x7f3be40b2800 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be9eba640 10 monclient(hunting): picked mon.noname-c con 0x7f3be40b38c0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be9eba640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be41e8250 con 0x7f3be40a9c00 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be9eba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be413adc0 con 0x7f3be40b2800 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be9eba640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be4070ca0 con 0x7f3be40b38c0 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be9eba640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be9eba640 10 monclient(hunting): authenticate will time out at 1165.205126s 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be3fff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f3be40b2800 0x7f3be41e7430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45173/0 (socket says 172.21.15.94:45173) 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.937+0000 7f3be3fff640 1 -- 172.21.15.94:0/3226881528 learned_addr learned my addr 172.21.15.94:0/3226881528 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:13.928 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 client.?.objecter ms_handle_connect 0x7f3be40b2800 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 client.?.objecter ms_handle_connect 0x7f3be40b38c0 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179936686 0 0) 0x7f3be413adc0 con 0x7f3be40b2800 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.939+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd0004eb0 con 0x7f3be40b2800 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547357153 0 0) 0x7f3be4070ca0 con 0x7f3be40b38c0 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.929 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be413adc0 con 0x7f3be40b38c0 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 client.?.objecter ms_handle_connect 0x7f3be40a9c00 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378866246 0 0) 0x7f3bd0004eb0 con 0x7f3be40b2800 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be4070ca0 con 0x7f3be40b2800 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1308332547 0 0) 0x7f3be413adc0 con 0x7f3be40b38c0 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd0004eb0 con 0x7f3be40b38c0 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188090714 0 0) 0x7f3be41e8250 con 0x7f3be40a9c00 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.940+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be413adc0 con 0x7f3be40a9c00 2024-03-29T10:45:13.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3bdc001d10 con 0x7f3be40b2800 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient(hunting): dump: 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:13.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3bd4001990 con 0x7f3be40b38c0 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient(hunting): dump: 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:13.932 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 290760510 0 0) 0x7f3be4070ca0 con 0x7f3be40b2800 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 >> v1:172.21.15.94:6790/0 conn(0x7f3be40b38c0 legacy=0x7f3be41e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 >> v1:172.21.15.94:6789/0 conn(0x7f3be40a9c00 legacy=0x7f3be4070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 1 monclient: found mon.b 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.941+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be406f860 con 0x7f3be40b2800 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 5 monclient: authenticate success, global_id 4432 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 10 monclient: _renew_subs 2024-03-29T10:45:13.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be406fb40 con 0x7f3be40b2800 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 10 monclient: _renew_subs 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be4070100 con 0x7f3be40b2800 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be9eba640 1 librados: init done 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bdc001e70 con 0x7f3be40b2800 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f3bdc004e10 con 0x7f3be40b2800 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.942+0000 7f3be17fa640 10 monclient: dump: 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:13.934 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:13.935 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:13.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.944+0000 7f3be27fc640 4 set_mon_vals no callback set 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.946+0000 7f3be9eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.946+0000 7f3be9eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.946+0000 7f3be9eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.946+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc4005080 con 0x7f3be40b2800 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.951+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f3bdc01ade0 con 0x7f3be40b2800 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.951+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f3bdc084d20 con 0x7f3be40b2800 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.951+0000 7f3be17fa640 10 client.4432.objecter ms_dispatch 0x7f3be4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.951+0000 7f3be17fa640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.951+0000 7f3be17fa640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.951+0000 7f3be17fa640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.952+0000 7f3be8eb8640 10 monclient: get_auth_request con 0x7f3bd006e200 auth_method 0 2024-03-29T10:45:13.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.952+0000 7f3be17fa640 10 client.4432.objecter ms_handle_connect 0x7f3bd006e200 2024-03-29T10:45:13.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.956+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bdc050430 con 0x7f3be40b2800 2024-03-29T10:45:13.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.956+0000 7f3be17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:13.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:13.956+0000 7f3be17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:14.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.379+0000 7f3be9eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:14.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.379+0000 7f3be9eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:14.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.379+0000 7f3be9eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:45:14.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.379+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bc4005330 con 0x7f3be40b2800 2024-03-29T10:45:14.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.380+0000 7f3be17fa640 1 -- 172.21.15.94:0/3226881528 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f3bdc052530 con 0x7f3be40b2800 2024-03-29T10:45:14.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.381+0000 7f3be17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:14.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.381+0000 7f3be17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:14.363 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:14.364 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:14.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.384+0000 7f3be9eba640 10 librados: watch_flush enter 2024-03-29T10:45:14.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 10 librados: watch_flush exit 2024-03-29T10:45:14.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T10:45:14.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 10 client.4432.objecter successfully canceled tick 2024-03-29T10:45:14.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 >> v1:172.21.15.94:6816/3829934808 conn(0x7f3bd006e200 legacy=0x7f3bd00706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:14.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 10 monclient: shutdown 2024-03-29T10:45:14.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:14.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 >> v1:172.21.15.173:6789/0 conn(0x7f3be40b2800 legacy=0x7f3be41e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:14.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 shutdown_connections 2024-03-29T10:45:14.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 >> 172.21.15.94:0/3226881528 conn(0x7f3be4066a00 msgr2=0x7f3be4140400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:14.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 shutdown_connections 2024-03-29T10:45:14.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.385+0000 7f3be9eba640 1 -- 172.21.15.94:0/3226881528 wait complete. 2024-03-29T10:45:14.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:14.386+0000 7f3be9eba640 1 librados: shutdown 2024-03-29T10:45:14.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:21.191 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:21.334 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.351+0000 7fa6cf91c640 10 monclient: get_monmap_and_config 2024-03-29T10:45:21.334 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.351+0000 7fa6cf91c640 10 monclient: build_initial_monmap 2024-03-29T10:45:21.334 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.351+0000 7fa6cf91c640 10 monclient: monmap: 2024-03-29T10:45:21.334 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:21.334 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:21.334 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:21.352575+0000 2024-03-29T10:45:21.334 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:21.352575+0000 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.352+0000 7fa6cf91c640 1 Processor -- start 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 1 -- start start 2024-03-29T10:45:21.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 10 monclient: init 2024-03-29T10:45:21.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:21.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:21.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c80b4790 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:21.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c80aea90 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:21.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c80b39f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:21.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80b0c60 con 0x7fa6c80b4790 2024-03-29T10:45:21.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c80af020 con 0x7fa6c80aea90 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c813fbd0 con 0x7fa6c80b39f0 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.353+0000 7fa6cf91c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.354+0000 7fa6cf91c640 10 monclient(hunting): authenticate will time out at 1172.622096s 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.354+0000 7fa6cd6b8640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa6c80b4790 0x7fa6c80b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42946/0 (socket says 172.21.15.94:42946) 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.354+0000 7fa6cd6b8640 1 -- 172.21.15.94:0/2131658955 learned_addr learned my addr 172.21.15.94:0/2131658955 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197095924 0 0) 0x7fa6c80b0c60 con 0x7fa6c80b4790 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:45:21.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b0003520 con 0x7fa6c80b4790 2024-03-29T10:45:21.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388310186 0 0) 0x7fa6b0003520 con 0x7fa6c80b4790 2024-03-29T10:45:21.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.355+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 >> v1:172.21.15.94:6790/0 conn(0x7fa6c80b39f0 legacy=0x7fa6c813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 >> v1:172.21.15.173:6789/0 conn(0x7fa6c80aea90 legacy=0x7fa6c80af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 1 monclient: found mon.noname-a 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c813fd60 con 0x7fa6c80b4790 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa6bc001670 con 0x7fa6c80b4790 2024-03-29T10:45:21.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6cf91c640 5 monclient: authenticate success, global_id 4452 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6cf91c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6c7fff640 10 monclient: dump: 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:21.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.357+0000 7fa6cf91c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:21.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6bc001840 con 0x7fa6c80b4790 2024-03-29T10:45:21.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:21.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6cf91c640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6cf91c640 4 set_mon_vals no callback set 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6c7fff640 1 -- 172.21.15.94:0/2131658955 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa6bc001c90 con 0x7fa6c80b4790 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.358+0000 7fa6c7fff640 10 monclient: dump: 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:21.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.359+0000 7fa6cf91c640 10 monclient: shutdown 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.359+0000 7fa6cf91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.359+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2131658955 >> v1:172.21.15.94:6789/0 conn(0x7fa6c80b4790 legacy=0x7fa6c80b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.359+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2131658955 shutdown_connections 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.359+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2131658955 >> 172.21.15.94:0/2131658955 conn(0x7fa6c8066a00 msgr2=0x7fa6c8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.359+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2131658955 shutdown_connections 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.359+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2131658955 wait complete. 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.360+0000 7fa6cf91c640 10 monclient: build_initial_monmap 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.360+0000 7fa6cf91c640 10 monclient: monmap: 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:21.345 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:21.346 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:21.346 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:21.346 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:21.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.360+0000 7fa6cf91c640 1 librados: starting msgr at 2024-03-29T10:45:21.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.360+0000 7fa6cf91c640 1 librados: starting objecter 2024-03-29T10:45:21.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.361+0000 7fa6cf91c640 1 Processor -- start 2024-03-29T10:45:21.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.361+0000 7fa6cf91c640 1 -- start start 2024-03-29T10:45:21.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.361+0000 7fa6cf91c640 1 librados: setting wanted keys 2024-03-29T10:45:21.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.361+0000 7fa6cf91c640 1 librados: calling monclient init 2024-03-29T10:45:21.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.361+0000 7fa6cf91c640 10 monclient: init 2024-03-29T10:45:21.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.361+0000 7fa6cf91c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.361+0000 7fa6cf91c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.362+0000 7fa6cf91c640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c80b39f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.362+0000 7fa6cf91c640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c80ae330 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.362+0000 7fa6cf91c640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c81e4320 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.362+0000 7fa6cf91c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c806eda0 con 0x7fa6c81e4320 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.362+0000 7fa6cf91c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c814e5b0 con 0x7fa6c80ae330 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.362+0000 7fa6cf91c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c806f8f0 con 0x7fa6c80b39f0 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.363+0000 7fa6cf91c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.363+0000 7fa6cf91c640 10 monclient(hunting): authenticate will time out at 1172.631126s 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.363+0000 7fa6cdeb9640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa6c81e4320 0x7fa6c81e6710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43174/0 (socket says 172.21.15.94:43174) 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.363+0000 7fa6cdeb9640 1 -- 172.21.15.94:0/2129213280 learned_addr learned my addr 172.21.15.94:0/2129213280 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.363+0000 7fa6c5ffb640 10 client.?.objecter ms_handle_connect 0x7fa6c81e4320 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264867807 0 0) 0x7fa6b8001390 con 0x7fa6c81e4320 2024-03-29T10:45:21.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b4004eb0 con 0x7fa6c81e4320 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 client.?.objecter ms_handle_connect 0x7fa6c80b39f0 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078826597 0 0) 0x7fa6c806f8f0 con 0x7fa6c80b39f0 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:21.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b8001390 con 0x7fa6c80b39f0 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1781667680 0 0) 0x7fa6b4004eb0 con 0x7fa6c81e4320 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.364+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6c806f8f0 con 0x7fa6c81e4320 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa6b8001ac0 con 0x7fa6c81e4320 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:21.354 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589589105 0 0) 0x7fa6b8001390 con 0x7fa6c80b39f0 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6b4004eb0 con 0x7fa6c80b39f0 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa6bc001b50 con 0x7fa6c80b39f0 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.355 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3140129140 0 0) 0x7fa6c806f8f0 con 0x7fa6c81e4320 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 >> v1:172.21.15.94:6790/0 conn(0x7fa6c80b39f0 legacy=0x7fa6c806e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.365+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 >> v1:172.21.15.173:6789/0 conn(0x7fa6c80ae330 legacy=0x7fa6c814bc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.366+0000 7fa6c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.366+0000 7fa6c5ffb640 1 monclient: found mon.a 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.366+0000 7fa6c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.366+0000 7fa6c5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.366+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c806fbd0 con 0x7fa6c81e4320 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.366+0000 7fa6c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.366+0000 7fa6c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:21.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 5 monclient: authenticate success, global_id 4455 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 10 monclient: _renew_subs 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c806feb0 con 0x7fa6c81e4320 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 10 monclient: _renew_subs 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c80705b0 con 0x7fa6c81e4320 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.367+0000 7fa6cf91c640 1 librados: init done 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6b8001c20 con 0x7fa6c81e4320 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fa6b80020d0 con 0x7fa6c81e4320 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6c5ffb640 10 monclient: dump: 2024-03-29T10:45:21.357 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6c77fe640 4 set_mon_vals no callback set 2024-03-29T10:45:21.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.368+0000 7fa6cf91c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:21.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.369+0000 7fa6cf91c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.370+0000 7fa6cf91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.370+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a8005080 con 0x7fa6c81e4320 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.370+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fa6b801abf0 con 0x7fa6c81e4320 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.371+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fa6b8084c20 con 0x7fa6c81e4320 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.371+0000 7fa6c5ffb640 10 client.4455.objecter ms_dispatch 0x7fa6c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.371+0000 7fa6c5ffb640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.371+0000 7fa6c5ffb640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.371+0000 7fa6c5ffb640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.371+0000 7fa6cd6b8640 10 monclient: get_auth_request con 0x7fa6b406df00 auth_method 0 2024-03-29T10:45:21.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.372+0000 7fa6c5ffb640 10 client.4455.objecter ms_handle_connect 0x7fa6b406df00 2024-03-29T10:45:21.361 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.379+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6b8050330 con 0x7fa6c81e4320 2024-03-29T10:45:21.361 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.379+0000 7fa6c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:21.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.379+0000 7fa6c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:21.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.695+0000 7fa6cf91c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:21.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.695+0000 7fa6cf91c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:21.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.695+0000 7fa6cf91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:21.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.695+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6a8005330 con 0x7fa6c81e4320 2024-03-29T10:45:21.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.696+0000 7fa6c5ffb640 1 -- 172.21.15.94:0/2129213280 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fa6b8052430 con 0x7fa6c81e4320 2024-03-29T10:45:21.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.696+0000 7fa6c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:21.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.696+0000 7fa6c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:21.679 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:21.679 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 10 librados: watch_flush enter 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 10 librados: watch_flush exit 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 10 client.4455.objecter successfully canceled tick 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 >> v1:172.21.15.94:6816/3829934808 conn(0x7fa6b406df00 legacy=0x7fa6b40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 10 monclient: shutdown 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:21.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.699+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 >> v1:172.21.15.94:6789/0 conn(0x7fa6c81e4320 legacy=0x7fa6c81e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:21.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.700+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 shutdown_connections 2024-03-29T10:45:21.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.700+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 >> 172.21.15.94:0/2129213280 conn(0x7fa6c8066a00 msgr2=0x7fa6c813f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:21.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.700+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 shutdown_connections 2024-03-29T10:45:21.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.700+0000 7fa6cf91c640 1 -- 172.21.15.94:0/2129213280 wait complete. 2024-03-29T10:45:21.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:21.700+0000 7fa6cf91c640 1 librados: shutdown 2024-03-29T10:45:21.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:28.506 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.650+0000 7f4daab1c640 10 monclient: get_monmap_and_config 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.650+0000 7f4daab1c640 10 monclient: build_initial_monmap 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.650+0000 7f4daab1c640 10 monclient: monmap: 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:28.651475+0000 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:28.651475+0000 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:28.633 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:28.634 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:28.634 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:28.634 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:28.634 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:28.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.651+0000 7f4daab1c640 1 Processor -- start 2024-03-29T10:45:28.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.651+0000 7f4daab1c640 1 -- start start 2024-03-29T10:45:28.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.651+0000 7f4daab1c640 10 monclient: init 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 10 monclient(hunting): picked mon.noname-a con 0x7f4da41422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 10 monclient(hunting): picked mon.noname-b con 0x7f4da4142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 10 monclient(hunting): picked mon.noname-c con 0x7f4da40b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da4142c80 con 0x7f4da41422b0 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da40aae40 con 0x7f4da40b17d0 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da413fbd0 con 0x7f4da4142690 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4daab1c640 10 monclient(hunting): authenticate will time out at 1179.920508s 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4da3fff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f4da4142690 0x7f4da40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:34134/0 (socket says 172.21.15.94:34134) 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4da3fff640 1 -- 172.21.15.94:0/2350021400 learned_addr learned my addr 172.21.15.94:0/2350021400 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:28.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602607198 0 0) 0x7f4da413fbd0 con 0x7f4da4142690 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4da37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4da37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.652+0000 7f4da37fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d94003520 con 0x7f4da4142690 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565210513 0 0) 0x7f4da4142c80 con 0x7f4da41422b0 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:45:28.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da413fbd0 con 0x7f4da41422b0 2024-03-29T10:45:28.641 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1591925370 0 0) 0x7f4da413fbd0 con 0x7f4da41422b0 2024-03-29T10:45:28.641 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.653+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 >> v1:172.21.15.94:6790/0 conn(0x7f4da4142690 legacy=0x7f4da40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.641 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.654+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 >> v1:172.21.15.173:6789/0 conn(0x7f4da40b17d0 legacy=0x7f4da40b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:28.641 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:28.641 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 1 monclient: found mon.noname-a 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da4151940 con 0x7f4da41422b0 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4d98001670 con 0x7f4da41422b0 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4daab1c640 5 monclient: authenticate success, global_id 4458 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4daab1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: dump: 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:28.642 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4daab1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d980017d0 con 0x7f4da41422b0 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 1 -- 172.21.15.94:0/2350021400 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4d98001ba0 con 0x7f4da41422b0 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:28.643 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.656+0000 7f4da37fe640 10 monclient: dump: 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.657+0000 7f4daab1c640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.657+0000 7f4daab1c640 4 set_mon_vals no callback set 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.657+0000 7f4daab1c640 10 monclient: shutdown 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.657+0000 7f4daab1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:28.644 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.657+0000 7f4daab1c640 1 -- 172.21.15.94:0/2350021400 >> v1:172.21.15.94:6789/0 conn(0x7f4da41422b0 legacy=0x7f4da413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.658+0000 7f4daab1c640 1 -- 172.21.15.94:0/2350021400 shutdown_connections 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.658+0000 7f4daab1c640 1 -- 172.21.15.94:0/2350021400 >> 172.21.15.94:0/2350021400 conn(0x7f4da4066a00 msgr2=0x7f4da4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.658+0000 7f4daab1c640 1 -- 172.21.15.94:0/2350021400 shutdown_connections 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.658+0000 7f4daab1c640 1 -- 172.21.15.94:0/2350021400 wait complete. 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 10 monclient: build_initial_monmap 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 10 monclient: monmap: 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:28.645 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:28.646 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:28.646 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:28.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 1 librados: starting msgr at 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 1 librados: starting objecter 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 1 Processor -- start 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 1 -- start start 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 1 librados: setting wanted keys 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 1 librados: calling monclient init 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.659+0000 7f4daab1c640 10 monclient: init 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.660+0000 7f4daab1c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.660+0000 7f4daab1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4daab1c640 10 monclient(hunting): picked mon.noname-c con 0x7f4da40b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4da88b8640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f4da40b17d0 0x7f4da40ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:34412/0 (socket says 172.21.15.94:34412) 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4da88b8640 1 -- 172.21.15.94:0/3341849280 learned_addr learned my addr 172.21.15.94:0/3341849280 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:28.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4daab1c640 10 monclient(hunting): picked mon.noname-a con 0x7f4da41422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4da17fa640 10 client.?.objecter ms_handle_connect 0x7f4da41422b0 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4da17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4da17fa640 10 client.?.objecter ms_handle_connect 0x7f4da40b17d0 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.661+0000 7f4da17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4daab1c640 10 monclient(hunting): picked mon.noname-b con 0x7f4da4142690 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da4131ef0 con 0x7f4da41422b0 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da40aca80 con 0x7f4da4142690 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da40acd60 con 0x7f4da40b17d0 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4daab1c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4daab1c640 10 monclient(hunting): authenticate will time out at 1179.930528s 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418360309 0 0) 0x7f4da4131ef0 con 0x7f4da41422b0 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:28.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d88004eb0 con 0x7f4da41422b0 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532352437 0 0) 0x7f4da40acd60 con 0x7f4da40b17d0 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.662+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da4131ef0 con 0x7f4da40b17d0 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3709774053 0 0) 0x7f4da4131ef0 con 0x7f4da40b17d0 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4da40acd60 con 0x7f4da40b17d0 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4d98001ae0 con 0x7f4da40b17d0 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient(hunting): dump: 2024-03-29T10:45:28.654 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 37765312 0 0) 0x7f4da40acd60 con 0x7f4da40b17d0 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 >> v1:172.21.15.173:6789/0 conn(0x7f4da4142690 legacy=0x7f4da4130110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 >> v1:172.21.15.94:6789/0 conn(0x7f4da41422b0 legacy=0x7f4da412fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:28.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 monclient: found mon.c 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da40ad040 con 0x7f4da40b17d0 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4da17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 10 monclient: _renew_subs 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da40abec0 con 0x7f4da40b17d0 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 10 monclient: _renew_subs 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da41e2e70 con 0x7f4da40b17d0 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.663+0000 7f4daab1c640 1 librados: init done 2024-03-29T10:45:28.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.664+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d980038a0 con 0x7f4da40b17d0 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.664+0000 7f4da17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.664+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4d98003cb0 con 0x7f4da40b17d0 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.664+0000 7f4da17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.664+0000 7f4da17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.664+0000 7f4da17fa640 10 monclient: dump: 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:28.657 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.664+0000 7f4da2ffd640 4 set_mon_vals no callback set 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.665+0000 7f4daab1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.665+0000 7f4daab1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.665+0000 7f4daab1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.665+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d84005080 con 0x7f4da40b17d0 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.666+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f4d98003ed0 con 0x7f4da40b17d0 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.666+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f4d98084b80 con 0x7f4da40b17d0 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.666+0000 7f4da17fa640 10 client.4475.objecter ms_dispatch 0x7f4da4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.666+0000 7f4da17fa640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.666+0000 7f4da17fa640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.667+0000 7f4da17fa640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.667+0000 7f4da3fff640 10 monclient: get_auth_request con 0x7f4d8806daf0 auth_method 0 2024-03-29T10:45:28.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.667+0000 7f4da17fa640 10 client.4475.objecter ms_handle_connect 0x7f4d8806daf0 2024-03-29T10:45:28.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.674+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d98050460 con 0x7f4da40b17d0 2024-03-29T10:45:28.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.674+0000 7f4da17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:28.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.674+0000 7f4da17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:28.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.959+0000 7f4daab1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:28.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.959+0000 7f4daab1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:28.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.959+0000 7f4daab1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:28.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.959+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d84005330 con 0x7f4da40b17d0 2024-03-29T10:45:28.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.960+0000 7f4da17fa640 1 -- 172.21.15.94:0/3341849280 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f4d98052390 con 0x7f4da40b17d0 2024-03-29T10:45:28.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.960+0000 7f4da17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:28.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.960+0000 7f4da17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:28.943 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:28.943 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.962+0000 7f4daab1c640 10 librados: watch_flush enter 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 10 librados: watch_flush exit 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 10 client.4475.objecter successfully canceled tick 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 >> v1:172.21.15.94:6816/3829934808 conn(0x7f4d8806daf0 legacy=0x7f4d8806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 10 monclient: shutdown 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 >> v1:172.21.15.94:6790/0 conn(0x7f4da40b17d0 legacy=0x7f4da40ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 shutdown_connections 2024-03-29T10:45:28.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 >> 172.21.15.94:0/3341849280 conn(0x7f4da4066a00 msgr2=0x7f4da413c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:28.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 shutdown_connections 2024-03-29T10:45:28.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 1 -- 172.21.15.94:0/3341849280 wait complete. 2024-03-29T10:45:28.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:28.963+0000 7f4daab1c640 1 librados: shutdown 2024-03-29T10:45:28.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:35.764 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:35.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.928+0000 7fe55cdc8640 10 monclient: get_monmap_and_config 2024-03-29T10:45:35.911 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.928+0000 7fe55cdc8640 10 monclient: build_initial_monmap 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.928+0000 7fe55cdc8640 10 monclient: monmap: 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:35.929917+0000 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:35.929917+0000 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:35.912 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:35.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.929+0000 7fe55cdc8640 1 Processor -- start 2024-03-29T10:45:35.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.929+0000 7fe55cdc8640 1 -- start start 2024-03-29T10:45:35.913 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.929+0000 7fe55cdc8640 10 monclient: init 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 10 monclient(hunting): picked mon.noname-a con 0x7fe5500389e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 10 monclient(hunting): picked mon.noname-b con 0x7fe550038dc0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 10 monclient(hunting): picked mon.noname-c con 0x7fe550030660 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe556ffd640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe550038dc0 0x7fe5500cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:57690/0 (socket says 172.21.15.94:57690) 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe556ffd640 1 -- 172.21.15.94:0/2322439747 learned_addr learned my addr 172.21.15.94:0/2322439747 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5500c54c0 con 0x7fe5500389e0 2024-03-29T10:45:35.918 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe550036430 con 0x7fe550030660 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.930+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe550039840 con 0x7fe550038dc0 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe55cdc8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe55cdc8640 10 monclient(hunting): authenticate will time out at 1187.200213s 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690709881 0 0) 0x7fe5500c54c0 con 0x7fe5500389e0 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe548003520 con 0x7fe5500389e0 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174024719 0 0) 0x7fe550039840 con 0x7fe550038dc0 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.919 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5500c54c0 con 0x7fe550038dc0 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3464938511 0 0) 0x7fe548003520 con 0x7fe5500389e0 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 >> v1:172.21.15.94:6790/0 conn(0x7fe550038dc0 legacy=0x7fe5500cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 >> v1:172.21.15.173:6789/0 conn(0x7fe550030660 legacy=0x7fe5500d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 1 monclient: found mon.noname-a 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.931+0000 7fe5567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe550035690 con 0x7fe5500389e0 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe54c001690 con 0x7fe5500389e0 2024-03-29T10:45:35.920 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe55cdc8640 5 monclient: authenticate success, global_id 4464 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe55cdc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: dump: 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:35.921 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe55cdc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe54c0017f0 con 0x7fe5500389e0 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 1 -- 172.21.15.94:0/2322439747 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe54c001c00 con 0x7fe5500389e0 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe55cdc8640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe55cdc8640 4 set_mon_vals no callback set 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.932+0000 7fe5567fc640 10 monclient: dump: 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.922 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.933+0000 7fe55cdc8640 10 monclient: shutdown 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.933+0000 7fe55cdc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.933+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 >> v1:172.21.15.94:6789/0 conn(0x7fe5500389e0 legacy=0x7fe550033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.933+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 shutdown_connections 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.933+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 >> 172.21.15.94:0/2322439747 conn(0x7fe550005c00 msgr2=0x7fe550006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.933+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 shutdown_connections 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.933+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2322439747 wait complete. 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.934+0000 7fe55cdc8640 10 monclient: build_initial_monmap 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.934+0000 7fe55cdc8640 10 monclient: monmap: 2024-03-29T10:45:35.923 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.934+0000 7fe55cdc8640 1 librados: starting msgr at 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.934+0000 7fe55cdc8640 1 librados: starting objecter 2024-03-29T10:45:35.924 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.935+0000 7fe55cdc8640 1 Processor -- start 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.935+0000 7fe55cdc8640 1 -- start start 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.935+0000 7fe55cdc8640 1 librados: setting wanted keys 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.935+0000 7fe55cdc8640 1 librados: calling monclient init 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.935+0000 7fe55cdc8640 10 monclient: init 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 10 monclient(hunting): picked mon.noname-a con 0x7fe550030660 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5500389e0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:35.930 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 10 monclient(hunting): picked mon.noname-b con 0x7fe55002f4c0 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5500a5990 con 0x7fe550030660 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55002f9e0 con 0x7fe55002f4c0 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55002ff00 con 0x7fe5500389e0 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe5577fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe550030660 0x7fe55015cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:50580/0 (socket says 172.21.15.94:50580) 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe5577fe640 1 -- 172.21.15.94:0/2147210378 learned_addr learned my addr 172.21.15.94:0/2147210378 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe55cdc8640 10 monclient(hunting): authenticate will time out at 1187.205807s 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe543fff640 10 client.?.objecter ms_handle_connect 0x7fe550030660 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe543fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe543fff640 10 client.?.objecter ms_handle_connect 0x7fe55002f4c0 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.936+0000 7fe543fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 client.?.objecter ms_handle_connect 0x7fe5500389e0 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:35.931 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403901292 0 0) 0x7fe5500a5990 con 0x7fe550030660 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c004eb0 con 0x7fe550030660 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620032637 0 0) 0x7fe55002f9e0 con 0x7fe55002f4c0 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5500a5990 con 0x7fe55002f4c0 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 582862268 0 0) 0x7fe53c004eb0 con 0x7fe550030660 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe55002f9e0 con 0x7fe550030660 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe54c001bc0 con 0x7fe550030660 2024-03-29T10:45:35.932 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): dump: 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:35.933 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754797109 0 0) 0x7fe55002ff00 con 0x7fe5500389e0 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.937+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c004eb0 con 0x7fe5500389e0 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2894619614 0 0) 0x7fe55002f9e0 con 0x7fe550030660 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 >> v1:172.21.15.94:6790/0 conn(0x7fe5500389e0 legacy=0x7fe5500c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 >> v1:172.21.15.173:6789/0 conn(0x7fe55002f4c0 legacy=0x7fe5501683f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 1 monclient: found mon.a 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55015d1e0 con 0x7fe550030660 2024-03-29T10:45:35.934 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 5 monclient: authenticate success, global_id 4467 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 10 monclient: _renew_subs 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55015d450 con 0x7fe550030660 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 10 monclient: _renew_subs 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5500c2db0 con 0x7fe550030660 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe55cdc8640 1 librados: init done 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe54c0038a0 con 0x7fe550030660 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe54c003cf0 con 0x7fe550030660 2024-03-29T10:45:35.935 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe543fff640 10 monclient: dump: 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.938+0000 7fe555ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.939+0000 7fe55cdc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:35.936 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.939+0000 7fe55cdc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.939+0000 7fe55cdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.939+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe538005080 con 0x7fe550030660 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.941+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fe54c003e50 con 0x7fe550030660 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.942+0000 7fe556ffd640 10 monclient: get_auth_request con 0x7fe53c06dd80 auth_method 0 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.942+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fe54c084b90 con 0x7fe550030660 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.942+0000 7fe543fff640 10 client.4467.objecter ms_dispatch 0x7fe558000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.942+0000 7fe543fff640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.942+0000 7fe543fff640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.942+0000 7fe543fff640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T10:45:35.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.942+0000 7fe543fff640 10 client.4467.objecter ms_handle_connect 0x7fe53c06dd80 2024-03-29T10:45:35.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.951+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe54c050470 con 0x7fe550030660 2024-03-29T10:45:35.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.951+0000 7fe543fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:35.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:35.951+0000 7fe543fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:36.288 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.305+0000 7fe55cdc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:36.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.305+0000 7fe55cdc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:36.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.305+0000 7fe55cdc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:45:36.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.305+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe538005330 con 0x7fe550030660 2024-03-29T10:45:36.289 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.306+0000 7fe543fff640 1 -- 172.21.15.94:0/2147210378 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fe54c0523a0 con 0x7fe550030660 2024-03-29T10:45:36.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.306+0000 7fe543fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:36.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.306+0000 7fe543fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:36.290 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:36.290 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:36.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.309+0000 7fe55cdc8640 10 librados: watch_flush enter 2024-03-29T10:45:36.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 10 librados: watch_flush exit 2024-03-29T10:45:36.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T10:45:36.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 10 client.4467.objecter successfully canceled tick 2024-03-29T10:45:36.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 >> v1:172.21.15.94:6816/3829934808 conn(0x7fe53c06dd80 legacy=0x7fe53c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:36.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 10 monclient: shutdown 2024-03-29T10:45:36.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:36.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 >> v1:172.21.15.94:6789/0 conn(0x7fe550030660 legacy=0x7fe55015cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:36.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 shutdown_connections 2024-03-29T10:45:36.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 >> 172.21.15.94:0/2147210378 conn(0x7fe550005c00 msgr2=0x7fe550034b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:36.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 shutdown_connections 2024-03-29T10:45:36.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 1 -- 172.21.15.94:0/2147210378 wait complete. 2024-03-29T10:45:36.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:36.310+0000 7fe55cdc8640 1 librados: shutdown 2024-03-29T10:45:36.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:43.115 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:43.247 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.265+0000 7f27144c9640 10 monclient: get_monmap_and_config 2024-03-29T10:45:43.247 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.265+0000 7f27144c9640 10 monclient: build_initial_monmap 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.265+0000 7f27144c9640 10 monclient: monmap: 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:43.265978+0000 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:43.265978+0000 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:43.248 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:43.249 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.266+0000 7f27144c9640 1 Processor -- start 2024-03-29T10:45:43.249 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.266+0000 7f27144c9640 1 -- start start 2024-03-29T10:45:43.249 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.266+0000 7f27144c9640 10 monclient: init 2024-03-29T10:45:43.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:43.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:43.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 10 monclient(hunting): picked mon.noname-b con 0x7f270c1422b0 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:43.250 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 10 monclient(hunting): picked mon.noname-a con 0x7f270c142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 10 monclient(hunting): picked mon.noname-c con 0x7f270c0b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c142c80 con 0x7f270c142690 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0aae40 con 0x7f270c0b17d0 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c13fbd0 con 0x7f270c1422b0 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f27144c9640 10 monclient(hunting): authenticate will time out at 1194.535414s 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2712265640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f270c1422b0 0x7f270c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36540/0 (socket says 172.21.15.94:36540) 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2712265640 1 -- 172.21.15.94:0/4012533849 learned_addr learned my addr 172.21.15.94:0/4012533849 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154111792 0 0) 0x7f270c13fbd0 con 0x7f270c1422b0 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 10 monclient(hunting): _negotiate 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:45:43.251 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f8003520 con 0x7f270c1422b0 2024-03-29T10:45:43.252 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789756765 0 0) 0x7f26f8003520 con 0x7f270c1422b0 2024-03-29T10:45:43.252 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 >> v1:172.21.15.173:6789/0 conn(0x7f270c0b17d0 legacy=0x7f270c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:43.252 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.267+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 >> v1:172.21.15.94:6789/0 conn(0x7f270c142690 legacy=0x7f270c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:43.253 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: _finish_hunting 0 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 1 monclient: found mon.noname-b 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c151940 con 0x7f270c1422b0 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: _finish_auth 0 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: _check_auth_tickets 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2708001690 con 0x7f270c1422b0 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: dump: 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:43.254 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f27144c9640 5 monclient: authenticate success, global_id 4484 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f27144c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27080018f0 con 0x7f270c1422b0 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 1 -- 172.21.15.94:0/4012533849 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2708001cc0 con 0x7f270c1422b0 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f27144c9640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:43.255 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f27144c9640 4 set_mon_vals no callback set 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.270+0000 7f2711263640 10 monclient: dump: 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.271+0000 7f27144c9640 10 monclient: shutdown 2024-03-29T10:45:43.256 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.271+0000 7f27144c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:43.257 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.271+0000 7f27144c9640 1 -- 172.21.15.94:0/4012533849 >> v1:172.21.15.94:6790/0 conn(0x7f270c1422b0 legacy=0x7f270c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.257 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.271+0000 7f27144c9640 1 -- 172.21.15.94:0/4012533849 shutdown_connections 2024-03-29T10:45:43.257 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.271+0000 7f27144c9640 1 -- 172.21.15.94:0/4012533849 >> 172.21.15.94:0/4012533849 conn(0x7f270c066a00 msgr2=0x7f270c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:43.257 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.271+0000 7f27144c9640 1 -- 172.21.15.94:0/4012533849 shutdown_connections 2024-03-29T10:45:43.257 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.271+0000 7f27144c9640 1 -- 172.21.15.94:0/4012533849 wait complete. 2024-03-29T10:45:43.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.272+0000 7f27144c9640 10 monclient: build_initial_monmap 2024-03-29T10:45:43.264 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.272+0000 7f27144c9640 10 monclient: monmap: 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.272+0000 7f27144c9640 1 librados: starting msgr at 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.272+0000 7f27144c9640 1 librados: starting objecter 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.273+0000 7f27144c9640 1 Processor -- start 2024-03-29T10:45:43.265 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.273+0000 7f27144c9640 1 -- start start 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.273+0000 7f27144c9640 1 librados: setting wanted keys 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.273+0000 7f27144c9640 1 librados: calling monclient init 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.273+0000 7f27144c9640 10 monclient: init 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 10 monclient(hunting): picked mon.noname-b con 0x7f270c0b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 10 monclient(hunting): picked mon.noname-a con 0x7f270c1422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2712a66640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f270c142690 0x7f270c1daaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36712/0 (socket says 172.21.15.94:36712) 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2712a66640 1 -- 172.21.15.94:0/1604769928 learned_addr learned my addr 172.21.15.94:0/1604769928 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 10 monclient(hunting): picked mon.noname-c con 0x7f270c142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0ab9b0 con 0x7f270c1422b0 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 client.?.objecter ms_handle_connect 0x7f270c142690 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:43.266 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0ad2a0 con 0x7f270c0b17d0 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270c0abe40 con 0x7f270c142690 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f27144c9640 10 monclient(hunting): authenticate will time out at 1194.542596s 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485353054 0 0) 0x7f270c0abe40 con 0x7f270c142690 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f0004eb0 con 0x7f270c142690 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 client.?.objecter ms_handle_connect 0x7f270c0b17d0 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000081551 0 0) 0x7f270c0ad2a0 con 0x7f270c0b17d0 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:43.267 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.274+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c0abe40 con 0x7f270c0b17d0 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 195330740 0 0) 0x7f270c0abe40 con 0x7f270c0b17d0 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f270c0ad2a0 con 0x7f270c0b17d0 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2708001c00 con 0x7f270c0b17d0 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:43.268 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:43.269 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:43.269 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:43.269 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.269 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4209111391 0 0) 0x7f26f0004eb0 con 0x7f270c142690 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f270c0abe40 con 0x7f270c142690 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2704004380 con 0x7f270c142690 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:43.270 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1347698955 0 0) 0x7f270c0abe40 con 0x7f270c142690 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 >> v1:172.21.15.173:6789/0 conn(0x7f270c0b17d0 legacy=0x7f270c13e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 >> v1:172.21.15.94:6789/0 conn(0x7f270c1422b0 legacy=0x7f270c0b0bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:43.271 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.275+0000 7f2702ffd640 1 monclient: found mon.c 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f2702ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f2702ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c0ab540 con 0x7f270c142690 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f2702ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f2702ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f2702ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 10 monclient: _renew_subs 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270c0ac070 con 0x7f270c142690 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 10 monclient: _renew_subs 2024-03-29T10:45:43.272 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f270c0b14b0 con 0x7f270c142690 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2704001250 con 0x7f270c142690 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.276+0000 7f27144c9640 1 librados: init done 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.277+0000 7f2702ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.277+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2704001620 con 0x7f270c142690 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.277+0000 7f2702ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.277+0000 7f2702ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.277+0000 7f2702ffd640 10 monclient: dump: 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:43.273 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.277+0000 7f2703fff640 4 set_mon_vals no callback set 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.278+0000 7f27144c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.278+0000 7f27144c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.278+0000 7f27144c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.278+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270c075f10 con 0x7f270c142690 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.278+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f2704001780 con 0x7f270c142690 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.279+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f2704084d40 con 0x7f270c142690 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.279+0000 7f2702ffd640 10 client.4487.objecter ms_dispatch 0x7f270c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.279+0000 7f2702ffd640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.279+0000 7f2702ffd640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:43.274 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.279+0000 7f2702ffd640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T10:45:43.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.279+0000 7f2712265640 10 monclient: get_auth_request con 0x7f26f006de90 auth_method 0 2024-03-29T10:45:43.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.280+0000 7f2702ffd640 10 client.4487.objecter ms_handle_connect 0x7f26f006de90 2024-03-29T10:45:43.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.286+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2704050680 con 0x7f270c142690 2024-03-29T10:45:43.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.286+0000 7f2702ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:43.275 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.286+0000 7f2702ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:43.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.559+0000 7f27144c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:43.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.559+0000 7f27144c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:43.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.559+0000 7f27144c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:43.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.559+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f270c0b2e20 con 0x7f270c142690 2024-03-29T10:45:43.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.559+0000 7f2702ffd640 1 -- 172.21.15.94:0/1604769928 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f27040843f0 con 0x7f270c142690 2024-03-29T10:45:43.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.559+0000 7f2702ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:43.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.559+0000 7f2702ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:43.543 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:43.543 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 10 librados: watch_flush enter 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 10 librados: watch_flush exit 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 10 client.4487.objecter successfully canceled tick 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 >> v1:172.21.15.94:6816/3829934808 conn(0x7f26f006de90 legacy=0x7f26f0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 10 monclient: shutdown 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 >> v1:172.21.15.94:6790/0 conn(0x7f270c142690 legacy=0x7f270c1daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 shutdown_connections 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 >> 172.21.15.94:0/1604769928 conn(0x7f270c066a00 msgr2=0x7f270c13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 shutdown_connections 2024-03-29T10:45:43.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 1 -- 172.21.15.94:0/1604769928 wait complete. 2024-03-29T10:45:43.547 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:43.563+0000 7f27144c9640 1 librados: shutdown 2024-03-29T10:45:43.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:50.366 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:50.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.532+0000 7f0686c25640 10 monclient: get_monmap_and_config 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.532+0000 7f0686c25640 10 monclient: build_initial_monmap 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.532+0000 7f0686c25640 10 monclient: monmap: 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:50.533232+0000 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:50.533232+0000 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:50.516 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:50.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.533+0000 7f0686c25640 1 Processor -- start 2024-03-29T10:45:50.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.534+0000 7f0686c25640 1 -- start start 2024-03-29T10:45:50.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.534+0000 7f0686c25640 10 monclient: init 2024-03-29T10:45:50.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.534+0000 7f0686c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:50.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:50.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 10 monclient(hunting): picked mon.noname-c con 0x7f06800b4100 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:50.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 10 monclient(hunting): picked mon.noname-b con 0x7f068013ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:50.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 10 monclient(hunting): picked mon.noname-a con 0x7f06800af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:50.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800afd10 con 0x7f06800af900 2024-03-29T10:45:50.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800af6f0 con 0x7f06800b4100 2024-03-29T10:45:50.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800aaf60 con 0x7f068013ee90 2024-03-29T10:45:50.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:50.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.535+0000 7f0686c25640 10 monclient(hunting): authenticate will time out at 1201.803202s 2024-03-29T10:45:50.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f068013ee90 0x7f06800adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:34216/0 (socket says 172.21.15.94:34216) 2024-03-29T10:45:50.521 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067ffff640 1 -- 172.21.15.94:0/1686768891 learned_addr learned my addr 172.21.15.94:0/1686768891 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362241054 0 0) 0x7f06800aaf60 con 0x7f068013ee90 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c003520 con 0x7f068013ee90 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1537351949 0 0) 0x7f066c003520 con 0x7f068013ee90 2024-03-29T10:45:50.522 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.537+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 >> v1:172.21.15.173:6789/0 conn(0x7f06800b4100 legacy=0x7f06800a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:50.523 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 >> v1:172.21.15.94:6789/0 conn(0x7f06800af900 legacy=0x7f068013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068013e970 con 0x7f068013ee90 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0668001690 con 0x7f068013ee90 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f067f7fe640 10 monclient: dump: 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f0686c25640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.540+0000 7f0686c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:50.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0668001860 con 0x7f068013ee90 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f067f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f067f7fe640 1 -- 172.21.15.94:0/1686768891 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0668001c70 con 0x7f068013ee90 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f067f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f067f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f067f7fe640 10 monclient: dump: 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.526 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:50.527 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:50.527 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:50.527 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:50.527 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:50.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f0686c25640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:50.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.541+0000 7f0686c25640 4 set_mon_vals no callback set 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.542+0000 7f0686c25640 10 monclient: shutdown 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.542+0000 7f0686c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.542+0000 7f0686c25640 1 -- 172.21.15.94:0/1686768891 >> v1:172.21.15.94:6790/0 conn(0x7f068013ee90 legacy=0x7f06800adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.543+0000 7f0686c25640 1 -- 172.21.15.94:0/1686768891 shutdown_connections 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.543+0000 7f0686c25640 1 -- 172.21.15.94:0/1686768891 >> 172.21.15.94:0/1686768891 conn(0x7f06800b35e0 msgr2=0x7f06800b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.543+0000 7f0686c25640 1 -- 172.21.15.94:0/1686768891 shutdown_connections 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.543+0000 7f0686c25640 1 -- 172.21.15.94:0/1686768891 wait complete. 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.544+0000 7f0686c25640 10 monclient: build_initial_monmap 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.544+0000 7f0686c25640 10 monclient: monmap: 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:50.529 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.544+0000 7f0686c25640 1 librados: starting msgr at 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.544+0000 7f0686c25640 1 librados: starting objecter 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.545+0000 7f0686c25640 1 Processor -- start 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.545+0000 7f0686c25640 1 -- start start 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.545+0000 7f0686c25640 1 librados: setting wanted keys 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.545+0000 7f0686c25640 1 librados: calling monclient init 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.545+0000 7f0686c25640 10 monclient: init 2024-03-29T10:45:50.530 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.546+0000 7f0686c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:50.531 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.546+0000 7f0686c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.546+0000 7f0686c25640 10 monclient(hunting): picked mon.noname-a con 0x7f06800af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.548+0000 7f0686c25640 10 monclient(hunting): picked mon.noname-b con 0x7f06800b4100 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.548+0000 7f0686c25640 10 monclient(hunting): picked mon.noname-c con 0x7f068013ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.548+0000 7f0686c25640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06801d6ac0 con 0x7f06800af900 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.548+0000 7f0686c25640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06801d6f50 con 0x7f06800b4100 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.548+0000 7f0686c25640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0680142290 con 0x7f068013ee90 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.548+0000 7f0686c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.548+0000 7f0686c25640 10 monclient(hunting): authenticate will time out at 1201.816285s 2024-03-29T10:45:50.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.549+0000 7f06851c2640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f068013ee90 0x7f068013c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:34990/0 (socket says 172.21.15.94:34990) 2024-03-29T10:45:50.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.549+0000 7f06851c2640 1 -- 172.21.15.94:0/1662451643 learned_addr learned my addr 172.21.15.94:0/1662451643 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:50.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.549+0000 7f067d7fa640 10 client.?.objecter ms_handle_connect 0x7f068013ee90 2024-03-29T10:45:50.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.549+0000 7f067d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:50.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.551+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214786701 0 0) 0x7f0680142290 con 0x7f068013ee90 2024-03-29T10:45:50.542 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0660004eb0 con 0x7f068013ee90 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 client.?.objecter ms_handle_connect 0x7f06800af900 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 client.?.objecter ms_handle_connect 0x7f06800b4100 2024-03-29T10:45:50.543 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2814982222 0 0) 0x7f0660004eb0 con 0x7f068013ee90 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0680142290 con 0x7f068013ee90 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0670001c30 con 0x7f068013ee90 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.552+0000 7f067d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 10 monclient(hunting): dump: 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:50.544 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1983998646 0 0) 0x7f0680142290 con 0x7f068013ee90 2024-03-29T10:45:50.545 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 >> v1:172.21.15.173:6789/0 conn(0x7f06800b4100 legacy=0x7f0680140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 >> v1:172.21.15.94:6789/0 conn(0x7f06800af900 legacy=0x7f06801358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:50.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 1 monclient: found mon.c 2024-03-29T10:45:50.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:50.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:50.546 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0680135fc0 con 0x7f068013ee90 2024-03-29T10:45:50.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:50.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:50.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f067d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:50.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f0686c25640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:45:50.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f0686c25640 10 monclient: _renew_subs 2024-03-29T10:45:50.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f0686c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:50.550 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.553+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0680136210 con 0x7f068013ee90 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0670001d90 con 0x7f068013ee90 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f0686c25640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f067d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0670003bb0 con 0x7f068013ee90 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f067e7fc640 4 set_mon_vals no callback set 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f067d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f067d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f067d7fa640 10 monclient: dump: 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:50.551 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f0686c25640 10 monclient: _renew_subs 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f0686c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f068013cf30 con 0x7f068013ee90 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.554+0000 7f0686c25640 1 librados: init done 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f0686c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f0670003d10 con 0x7f068013ee90 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f0686c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f0686c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0680075f10 con 0x7f068013ee90 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0670084da0 con 0x7f068013ee90 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f067d7fa640 10 client.4493.objecter ms_dispatch 0x7f0680000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:50.552 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f067d7fa640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:50.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f067d7fa640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:50.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.556+0000 7f067d7fa640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T10:45:50.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.557+0000 7f06849c1640 10 monclient: get_auth_request con 0x7f066006dd50 auth_method 0 2024-03-29T10:45:50.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.557+0000 7f067d7fa640 10 client.4493.objecter ms_handle_connect 0x7f066006dd50 2024-03-29T10:45:50.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.566+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06700504b0 con 0x7f068013ee90 2024-03-29T10:45:50.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.566+0000 7f067d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:50.553 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.566+0000 7f067d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:50.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.895+0000 7f0686c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:50.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.895+0000 7f0686c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:50.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.895+0000 7f0686c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:50.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.895+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f068006ab70 con 0x7f068013ee90 2024-03-29T10:45:50.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.897+0000 7f067d7fa640 1 -- 172.21.15.94:0/1662451643 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f06700525b0 con 0x7f068013ee90 2024-03-29T10:45:50.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.897+0000 7f067d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:50.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.897+0000 7f067d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:50.880 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:50.880 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:50.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 10 librados: watch_flush enter 2024-03-29T10:45:50.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 10 librados: watch_flush exit 2024-03-29T10:45:50.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T10:45:50.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 10 client.4493.objecter successfully canceled tick 2024-03-29T10:45:50.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 >> v1:172.21.15.94:6816/3829934808 conn(0x7f066006dd50 legacy=0x7f0660070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 10 monclient: shutdown 2024-03-29T10:45:50.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:50.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.901+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 >> v1:172.21.15.94:6790/0 conn(0x7f068013ee90 legacy=0x7f068013c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:50.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.902+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 shutdown_connections 2024-03-29T10:45:50.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.902+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 >> 172.21.15.94:0/1662451643 conn(0x7f06800b35e0 msgr2=0x7f06800adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:50.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.902+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 shutdown_connections 2024-03-29T10:45:50.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.902+0000 7f0686c25640 1 -- 172.21.15.94:0/1662451643 wait complete. 2024-03-29T10:45:50.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:50.902+0000 7f0686c25640 1 librados: shutdown 2024-03-29T10:45:50.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:57.708 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:57.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.873+0000 7f5d40a24640 10 monclient: get_monmap_and_config 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.873+0000 7f5d40a24640 10 monclient: build_initial_monmap 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.873+0000 7f5d40a24640 10 monclient: monmap: 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:45:57.874577+0000 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:45:57.874577+0000 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:45:57.857 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:57.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.874+0000 7f5d40a24640 1 Processor -- start 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.874+0000 7f5d40a24640 1 -- start start 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.874+0000 7f5d40a24640 10 monclient: init 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.875+0000 7f5d40a24640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.875+0000 7f5d40a24640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.875+0000 7f5d40a24640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3c14a400 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.875+0000 7f5d40a24640 10 monclient(hunting): picked mon.noname-c con 0x7f5d3c0aa500 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.878+0000 7f5d3b7fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f5d3c14a400 0x7f5d3c0aac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:40130/0 (socket says 172.21.15.94:40130) 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.878+0000 7f5d3b7fe640 1 -- 172.21.15.94:0/4269602051 learned_addr learned my addr 172.21.15.94:0/4269602051 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.878+0000 7f5d40a24640 10 monclient(hunting): picked mon.noname-a con 0x7f5d3c149800 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.879+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0aca50 con 0x7f5d3c149800 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.879+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c149cb0 con 0x7f5d3c0aa500 2024-03-29T10:45:57.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.879+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0acc80 con 0x7f5d3c14a400 2024-03-29T10:45:57.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.879+0000 7f5d40a24640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:57.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.879+0000 7f5d40a24640 10 monclient(hunting): authenticate will time out at 1209.147430s 2024-03-29T10:45:57.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217469635 0 0) 0x7f5d3c0aca50 con 0x7f5d3c149800 2024-03-29T10:45:57.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:57.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:57.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d24003520 con 0x7f5d3c149800 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997675926 0 0) 0x7f5d3c0acc80 con 0x7f5d3c14a400 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.881+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c0aca50 con 0x7f5d3c14a400 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2282504584 0 0) 0x7f5d24003520 con 0x7f5d3c149800 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 >> v1:172.21.15.94:6790/0 conn(0x7f5d3c14a400 legacy=0x7f5d3c0aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 >> v1:172.21.15.173:6789/0 conn(0x7f5d3c0aa500 legacy=0x7f5d3c0ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:57.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0ac350 con 0x7f5d3c149800 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5d34001a10 con 0x7f5d3c149800 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d40a24640 5 monclient: authenticate success, global_id 4476 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.882+0000 7f5d40a24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 10 monclient: dump: 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:57.869 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:57.870 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d40a24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:57.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d34001c30 con 0x7f5d3c149800 2024-03-29T10:45:57.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:57.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 1 -- 172.21.15.94:0/4269602051 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5d34001fc0 con 0x7f5d3c149800 2024-03-29T10:45:57.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:57.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d3a7fc640 10 monclient: dump: 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d40a24640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.883+0000 7f5d40a24640 4 set_mon_vals no callback set 2024-03-29T10:45:57.872 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.884+0000 7f5d40a24640 10 monclient: shutdown 2024-03-29T10:45:57.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.884+0000 7f5d40a24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:57.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.884+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 >> v1:172.21.15.94:6789/0 conn(0x7f5d3c149800 legacy=0x7f5d3c151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:57.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.885+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 shutdown_connections 2024-03-29T10:45:57.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.885+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 >> 172.21.15.94:0/4269602051 conn(0x7f5d3c0b3a30 msgr2=0x7f5d3c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:57.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.888+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 shutdown_connections 2024-03-29T10:45:57.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.889+0000 7f5d40a24640 1 -- 172.21.15.94:0/4269602051 wait complete. 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.890+0000 7f5d40a24640 10 monclient: build_initial_monmap 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.890+0000 7f5d40a24640 10 monclient: monmap: 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:45:57.875 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:57.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.891+0000 7f5d40a24640 1 librados: starting msgr at 2024-03-29T10:45:57.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.891+0000 7f5d40a24640 1 librados: starting objecter 2024-03-29T10:45:57.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.891+0000 7f5d40a24640 1 Processor -- start 2024-03-29T10:45:57.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.891+0000 7f5d40a24640 1 -- start start 2024-03-29T10:45:57.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.891+0000 7f5d40a24640 1 librados: setting wanted keys 2024-03-29T10:45:57.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.891+0000 7f5d40a24640 1 librados: calling monclient init 2024-03-29T10:45:57.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.891+0000 7f5d40a24640 10 monclient: init 2024-03-29T10:45:57.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.892+0000 7f5d40a24640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:57.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:57.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3c0aa500 addr v1:172.21.15.173:6789/0 2024-03-29T10:45:57.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 10 monclient(hunting): picked mon.noname-a con 0x7f5d3c149800 addr v1:172.21.15.94:6789/0 2024-03-29T10:45:57.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 10 monclient(hunting): picked mon.noname-c con 0x7f5d3c14a400 addr v1:172.21.15.94:6790/0 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c1db370 con 0x7f5d3c149800 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c1dc240 con 0x7f5d3c0aa500 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c1db610 con 0x7f5d3c14a400 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d40a24640 10 monclient(hunting): authenticate will time out at 1209.161321s 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d3bfff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f5d3c14a400 0x7f5d3c137570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41634/0 (socket says 172.21.15.94:41634) 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.893+0000 7f5d3bfff640 1 -- 172.21.15.94:0/121010400 learned_addr learned my addr 172.21.15.94:0/121010400 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.896+0000 7f5d23fff640 10 client.?.objecter ms_handle_connect 0x7f5d3c14a400 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.896+0000 7f5d23fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.896+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738641426 0 0) 0x7f5d3c1db610 con 0x7f5d3c14a400 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.896+0000 7f5d23fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.896+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:57.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.896+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.896+0000 7f5d23fff640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d28004eb0 con 0x7f5d3c14a400 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 client.?.objecter ms_handle_connect 0x7f5d3c149800 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 client.?.objecter ms_handle_connect 0x7f5d3c0aa500 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377434120 0 0) 0x7f5d3c1db370 con 0x7f5d3c149800 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c1db610 con 0x7f5d3c149800 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.897+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135916454 0 0) 0x7f5d3c1dc240 con 0x7f5d3c0aa500 2024-03-29T10:45:57.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c1db370 con 0x7f5d3c0aa500 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978057198 0 0) 0x7f5d28004eb0 con 0x7f5d3c14a400 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d3c1dc240 con 0x7f5d3c14a400 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5d34001a10 con 0x7f5d3c14a400 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): dump: 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:57.884 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:57.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.898+0000 7f5d23fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:57.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.900+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1572682487 0 0) 0x7f5d3c1dc240 con 0x7f5d3c14a400 2024-03-29T10:45:57.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 >> v1:172.21.15.173:6789/0 conn(0x7f5d3c0aa500 legacy=0x7f5d3c1dbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:57.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 >> v1:172.21.15.94:6789/0 conn(0x7f5d3c149800 legacy=0x7f5d3c136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:57.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:57.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 1 monclient: found mon.c 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c144ed0 con 0x7f5d3c14a400 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 10 monclient: _renew_subs 2024-03-29T10:45:57.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c145140 con 0x7f5d3c14a400 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 10 monclient: _renew_subs 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c145dc0 con 0x7f5d3c14a400 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d40a24640 1 librados: init done 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d34003c20 con 0x7f5d3c14a400 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.901+0000 7f5d39ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.902+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5d34004030 con 0x7f5d3c14a400 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.902+0000 7f5d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.902+0000 7f5d23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.902+0000 7f5d23fff640 10 monclient: dump: 2024-03-29T10:45:57.888 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.903+0000 7f5d40a24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.904+0000 7f5d40a24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.904+0000 7f5d40a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:57.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.904+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c04f730 con 0x7f5d3c14a400 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.906+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f5d34004190 con 0x7f5d3c14a400 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.907+0000 7f5d3b7fe640 10 monclient: get_auth_request con 0x7f5d2806ddb0 auth_method 0 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.907+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f5d34084eb0 con 0x7f5d3c14a400 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.907+0000 7f5d23fff640 10 client.4499.objecter ms_dispatch 0x7f5d3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.907+0000 7f5d23fff640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.907+0000 7f5d23fff640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.907+0000 7f5d23fff640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:45:57.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.907+0000 7f5d23fff640 10 client.4499.objecter ms_handle_connect 0x7f5d2806ddb0 2024-03-29T10:45:57.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.919+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d34050790 con 0x7f5d3c14a400 2024-03-29T10:45:57.902 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.919+0000 7f5d23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:57.903 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:57.919+0000 7f5d23fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:58.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.245+0000 7f5d40a24640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:58.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.245+0000 7f5d40a24640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:58.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.245+0000 7f5d40a24640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:45:58.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.245+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d3c074d80 con 0x7f5d3c14a400 2024-03-29T10:45:58.228 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:45:58.228 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:58.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.246+0000 7f5d23fff640 1 -- 172.21.15.94:0/121010400 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f5d340526c0 con 0x7f5d3c14a400 2024-03-29T10:45:58.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.246+0000 7f5d23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:58.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.246+0000 7f5d23fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:58.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.249+0000 7f5d40a24640 10 librados: watch_flush enter 2024-03-29T10:45:58.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.249+0000 7f5d40a24640 10 librados: watch_flush exit 2024-03-29T10:45:58.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.249+0000 7f5d40a24640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:45:58.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.249+0000 7f5d40a24640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:45:58.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 >> v1:172.21.15.94:6816/3829934808 conn(0x7f5d2806ddb0 legacy=0x7f5d28070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 10 monclient: shutdown 2024-03-29T10:45:58.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:58.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 >> v1:172.21.15.94:6790/0 conn(0x7f5d3c14a400 legacy=0x7f5d3c137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:58.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 shutdown_connections 2024-03-29T10:45:58.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 >> 172.21.15.94:0/121010400 conn(0x7f5d3c0b3a30 msgr2=0x7f5d3c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:58.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 shutdown_connections 2024-03-29T10:45:58.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.251+0000 7f5d40a24640 1 -- 172.21.15.94:0/121010400 wait complete. 2024-03-29T10:45:58.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:45:58.252+0000 7f5d40a24640 1 librados: shutdown 2024-03-29T10:45:58.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:05.059 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.229+0000 7ff82911e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.229+0000 7ff82911e640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.229+0000 7ff82911e640 10 monclient: monmap: 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:05.230318+0000 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:05.230318+0000 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:05.212 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:05.213 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:05.213 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:05.213 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:05.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.230+0000 7ff82911e640 1 Processor -- start 2024-03-29T10:46:05.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.230+0000 7ff82911e640 1 -- start start 2024-03-29T10:46:05.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.230+0000 7ff82911e640 10 monclient: init 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 10 monclient(hunting): picked mon.noname-c con 0x7ff8241457f0 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 10 monclient(hunting): picked mon.noname-b con 0x7ff82413f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 10 monclient(hunting): picked mon.noname-a con 0x7ff8240a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240aaef0 con 0x7ff8240a9d70 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff824145bd0 con 0x7ff8241457f0 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff824145eb0 con 0x7ff82413f650 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.231+0000 7ff82911e640 10 monclient(hunting): authenticate will time out at 1216.499330s 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.232+0000 7ff82259c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff82413f650 0x7ff8240b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:39842/0 (socket says 172.21.15.94:39842) 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.232+0000 7ff82259c640 1 -- 172.21.15.94:0/1906523282 learned_addr learned my addr 172.21.15.94:0/1906523282 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.233+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836199694 0 0) 0x7ff824145eb0 con 0x7ff82413f650 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.233+0000 7ff821d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.233+0000 7ff821d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.233+0000 7ff821d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.233+0000 7ff821d9b640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.233+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c003520 con 0x7ff82413f650 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.233+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4099741628 0 0) 0x7ff80c003520 con 0x7ff82413f650 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 >> v1:172.21.15.173:6789/0 conn(0x7ff8241457f0 legacy=0x7ff82413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 >> v1:172.21.15.94:6789/0 conn(0x7ff8240a9d70 legacy=0x7ff82414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:05.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 1 monclient: found mon.noname-b 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff824146340 con 0x7ff82413f650 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff814001690 con 0x7ff82413f650 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff821d9b640 10 monclient: dump: 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:05.219 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff82911e640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:46:05.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.234+0000 7ff82911e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:05.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8140018f0 con 0x7ff82413f650 2024-03-29T10:46:05.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff821d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff82911e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff82911e640 4 set_mon_vals no callback set 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff821d9b640 1 -- 172.21.15.94:0/1906523282 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff814001d40 con 0x7ff82413f650 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff821d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff821d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.237+0000 7ff821d9b640 10 monclient: dump: 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:05.222 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.238+0000 7ff82911e640 10 monclient: shutdown 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.238+0000 7ff82911e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.238+0000 7ff82911e640 1 -- 172.21.15.94:0/1906523282 >> v1:172.21.15.94:6790/0 conn(0x7ff82413f650 legacy=0x7ff8240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.238+0000 7ff82911e640 1 -- 172.21.15.94:0/1906523282 shutdown_connections 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.238+0000 7ff82911e640 1 -- 172.21.15.94:0/1906523282 >> 172.21.15.94:0/1906523282 conn(0x7ff824066a00 msgr2=0x7ff824066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.238+0000 7ff82911e640 1 -- 172.21.15.94:0/1906523282 shutdown_connections 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.238+0000 7ff82911e640 1 -- 172.21.15.94:0/1906523282 wait complete. 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.239+0000 7ff82911e640 10 monclient: build_initial_monmap 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.239+0000 7ff82911e640 10 monclient: monmap: 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:05.223 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:05.224 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:05.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.240+0000 7ff82911e640 1 librados: starting msgr at 2024-03-29T10:46:05.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.240+0000 7ff82911e640 1 librados: starting objecter 2024-03-29T10:46:05.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.240+0000 7ff82911e640 1 Processor -- start 2024-03-29T10:46:05.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.240+0000 7ff82911e640 1 -- start start 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.240+0000 7ff82911e640 1 librados: setting wanted keys 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.240+0000 7ff82911e640 1 librados: calling monclient init 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.240+0000 7ff82911e640 10 monclient: init 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 10 monclient(hunting): picked mon.noname-b con 0x7ff8240a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 10 monclient(hunting): picked mon.noname-c con 0x7ff82413f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 10 monclient(hunting): picked mon.noname-a con 0x7ff8241457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240711f0 con 0x7ff8241457f0 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82406ea60 con 0x7ff8240a9d70 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff824142ac0 con 0x7ff82413f650 2024-03-29T10:46:05.227 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:05.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.241+0000 7ff82911e640 10 monclient(hunting): authenticate will time out at 1216.509609s 2024-03-29T10:46:05.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.242+0000 7ff82259c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff82413f650 0x7ff8241423b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:52904/0 (socket says 172.21.15.94:52904) 2024-03-29T10:46:05.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.242+0000 7ff82259c640 1 -- 172.21.15.94:0/1471991970 learned_addr learned my addr 172.21.15.94:0/1471991970 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:05.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.242+0000 7ff80b7fe640 10 client.?.objecter ms_handle_connect 0x7ff82413f650 2024-03-29T10:46:05.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.242+0000 7ff80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.245+0000 7ff80b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8240a9d70 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.245+0000 7ff80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.245+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049566682 0 0) 0x7ff824142ac0 con 0x7ff82413f650 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff810004eb0 con 0x7ff82413f650 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048856755 0 0) 0x7ff82406ea60 con 0x7ff8240a9d70 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff824142ac0 con 0x7ff8240a9d70 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8241457f0 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028262723 0 0) 0x7ff810004eb0 con 0x7ff82413f650 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff82406ea60 con 0x7ff82413f650 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006874842 0 0) 0x7ff824142ac0 con 0x7ff8240a9d70 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff810004eb0 con 0x7ff8240a9d70 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff8140014b0 con 0x7ff82413f650 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): dump: 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:05.234 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:05.235 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.235 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.235 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.235 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:05.235 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:05.235 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493312558 0 0) 0x7ff81c003b50 con 0x7ff8241457f0 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:05.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff824142ac0 con 0x7ff8241457f0 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff818001690 con 0x7ff8240a9d70 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient(hunting): dump: 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:05.237 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1938805532 0 0) 0x7ff82406ea60 con 0x7ff82413f650 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 >> v1:172.21.15.173:6789/0 conn(0x7ff8240a9d70 legacy=0x7ff824144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 >> v1:172.21.15.94:6789/0 conn(0x7ff8241457f0 legacy=0x7ff8241e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 1 monclient: found mon.c 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.246+0000 7ff80b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82406ecd0 con 0x7ff82413f650 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff80b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff80b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:05.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff80b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 10 monclient: _renew_subs 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82406efb0 con 0x7ff82413f650 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 10 monclient: _renew_subs 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8241373c0 con 0x7ff82413f650 2024-03-29T10:46:05.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.247+0000 7ff82911e640 1 librados: init done 2024-03-29T10:46:05.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.249+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff814004c70 con 0x7ff82413f650 2024-03-29T10:46:05.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.249+0000 7ff80b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:05.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.249+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff814001610 con 0x7ff82413f650 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.249+0000 7ff80b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.249+0000 7ff80b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.249+0000 7ff80b7fe640 10 monclient: dump: 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:05.242 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.249+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7ff81401ae70 con 0x7ff82413f650 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff82159a640 4 set_mon_vals no callback set 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff822d9d640 10 monclient: get_auth_request con 0x7ff81006e1d0 auth_method 0 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff82911e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff82911e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff82911e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7ff814084cc0 con 0x7ff82413f650 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff804005080 con 0x7ff82413f650 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff80b7fe640 10 client.4505.objecter ms_dispatch 0x7ff824000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff80b7fe640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.250+0000 7ff80b7fe640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.251+0000 7ff80b7fe640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T10:46:05.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.251+0000 7ff80b7fe640 10 client.4505.objecter ms_handle_connect 0x7ff81006e1d0 2024-03-29T10:46:05.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.261+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8140506f0 con 0x7ff82413f650 2024-03-29T10:46:05.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.261+0000 7ff80b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:05.244 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.261+0000 7ff80b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:05.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.588+0000 7ff82911e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:05.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.588+0000 7ff82911e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:05.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.588+0000 7ff82911e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:05.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.588+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff804005330 con 0x7ff82413f650 2024-03-29T10:46:05.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.592+0000 7ff80b7fe640 1 -- 172.21.15.94:0/1471991970 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7ff814052620 con 0x7ff82413f650 2024-03-29T10:46:05.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.593+0000 7ff80b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:05.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.593+0000 7ff80b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:05.576 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:05.576 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:05.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.598+0000 7ff82911e640 10 librados: watch_flush enter 2024-03-29T10:46:05.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 10 librados: watch_flush exit 2024-03-29T10:46:05.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T10:46:05.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 10 client.4505.objecter successfully canceled tick 2024-03-29T10:46:05.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 >> v1:172.21.15.94:6816/3829934808 conn(0x7ff81006e1d0 legacy=0x7ff810070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 10 monclient: shutdown 2024-03-29T10:46:05.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:05.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 >> v1:172.21.15.94:6790/0 conn(0x7ff82413f650 legacy=0x7ff8241423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:05.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.599+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 shutdown_connections 2024-03-29T10:46:05.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.600+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 >> 172.21.15.94:0/1471991970 conn(0x7ff824066a00 msgr2=0x7ff8240b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:05.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.603+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 shutdown_connections 2024-03-29T10:46:05.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.603+0000 7ff82911e640 1 -- 172.21.15.94:0/1471991970 wait complete. 2024-03-29T10:46:05.585 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:05.603+0000 7ff82911e640 1 librados: shutdown 2024-03-29T10:46:05.602 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:12.409 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:12.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.575+0000 7f43e6493640 10 monclient: get_monmap_and_config 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.575+0000 7f43e6493640 10 monclient: build_initial_monmap 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.575+0000 7f43e6493640 10 monclient: monmap: 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:12.576234+0000 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:12.576234+0000 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:12.561 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:12.562 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:12.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.579+0000 7f43e6493640 1 Processor -- start 2024-03-29T10:46:12.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.579+0000 7f43e6493640 1 -- start start 2024-03-29T10:46:12.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.579+0000 7f43e6493640 10 monclient: init 2024-03-29T10:46:12.563 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 10 monclient(hunting): picked mon.noname-c con 0x7f43e00b4100 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 10 monclient(hunting): picked mon.noname-b con 0x7f43e013ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 10 monclient(hunting): picked mon.noname-a con 0x7f43e00a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e00af6f0 con 0x7f43e00a9ac0 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e00aac80 con 0x7f43e00b4100 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e00aaf60 con 0x7f43e013ee90 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:12.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.580+0000 7f43e6493640 10 monclient(hunting): authenticate will time out at 1223.848576s 2024-03-29T10:46:12.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43dffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f43e00b4100 0x7f43e00adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:46603/0 (socket says 172.21.15.94:46603) 2024-03-29T10:46:12.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43dffff640 1 -- 172.21.15.94:0/500419047 learned_addr learned my addr 172.21.15.94:0/500419047 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:12.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579769686 0 0) 0x7f43e00aac80 con 0x7f43e00b4100 2024-03-29T10:46:12.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:12.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:12.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43deffd640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.583+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c8003520 con 0x7f43e00b4100 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695870402 0 0) 0x7f43e00aaf60 con 0x7f43e013ee90 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e00aac80 con 0x7f43e013ee90 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1932664224 0 0) 0x7f43c8003520 con 0x7f43e00b4100 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 >> v1:172.21.15.94:6790/0 conn(0x7f43e013ee90 legacy=0x7f43e013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:12.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.584+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 >> v1:172.21.15.94:6789/0 conn(0x7f43e00a9ac0 legacy=0x7f43e014ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:12.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:12.572 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 1 monclient: found mon.noname-c 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e013e970 con 0x7f43e00b4100 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f43d4001690 con 0x7f43e00b4100 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.590+0000 7f43deffd640 10 monclient: dump: 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:12.573 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:12.574 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:12.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.591+0000 7f43e6493640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:46:12.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.591+0000 7f43e6493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:12.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43d40017f0 con 0x7f43e00b4100 2024-03-29T10:46:12.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:12.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43e6493640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:12.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43e6493640 4 set_mon_vals no callback set 2024-03-29T10:46:12.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43deffd640 1 -- 172.21.15.94:0/500419047 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f43d4001c40 con 0x7f43e00b4100 2024-03-29T10:46:12.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:12.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.592+0000 7f43deffd640 10 monclient: dump: 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.593+0000 7f43e6493640 10 monclient: shutdown 2024-03-29T10:46:12.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.593+0000 7f43e6493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:12.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.593+0000 7f43e6493640 1 -- 172.21.15.94:0/500419047 >> v1:172.21.15.173:6789/0 conn(0x7f43e00b4100 legacy=0x7f43e00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:12.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.593+0000 7f43e6493640 1 -- 172.21.15.94:0/500419047 shutdown_connections 2024-03-29T10:46:12.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.593+0000 7f43e6493640 1 -- 172.21.15.94:0/500419047 >> 172.21.15.94:0/500419047 conn(0x7f43e00b35e0 msgr2=0x7f43e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:12.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.593+0000 7f43e6493640 1 -- 172.21.15.94:0/500419047 shutdown_connections 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.593+0000 7f43e6493640 1 -- 172.21.15.94:0/500419047 wait complete. 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.594+0000 7f43e6493640 10 monclient: build_initial_monmap 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.594+0000 7f43e6493640 10 monclient: monmap: 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:12.581 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.594+0000 7f43e6493640 1 librados: starting msgr at 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.594+0000 7f43e6493640 1 librados: starting objecter 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.594+0000 7f43e6493640 1 Processor -- start 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.595+0000 7f43e6493640 1 -- start start 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.595+0000 7f43e6493640 1 librados: setting wanted keys 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.595+0000 7f43e6493640 1 librados: calling monclient init 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.595+0000 7f43e6493640 10 monclient: init 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 10 monclient(hunting): picked mon.noname-a con 0x7f43e00a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:12.582 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 10 monclient(hunting): picked mon.noname-b con 0x7f43e00b4100 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:12.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 10 monclient(hunting): picked mon.noname-c con 0x7f43e013ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:12.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e01df020 con 0x7f43e00a9ac0 2024-03-29T10:46:12.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e01dfbc0 con 0x7f43e00b4100 2024-03-29T10:46:12.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e01decb0 con 0x7f43e013ee90 2024-03-29T10:46:12.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:12.583 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.596+0000 7f43e6493640 10 monclient(hunting): authenticate will time out at 1223.864219s 2024-03-29T10:46:12.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.602+0000 7f43df7fe640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f43e00b4100 0x7f43e0071c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:46695/0 (socket says 172.21.15.94:46695) 2024-03-29T10:46:12.584 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.602+0000 7f43df7fe640 1 -- 172.21.15.94:0/3189788798 learned_addr learned my addr 172.21.15.94:0/3189788798 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 client.?.objecter ms_handle_connect 0x7f43e00b4100 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 client.?.objecter ms_handle_connect 0x7f43e00a9ac0 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653177861 0 0) 0x7f43e01dfbc0 con 0x7f43e00b4100 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:12.587 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc004eb0 con 0x7f43e00b4100 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098771702 0 0) 0x7f43e01df020 con 0x7f43e00a9ac0 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): my global_id is 4488 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e01dfbc0 con 0x7f43e00a9ac0 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 651128148 0 0) 0x7f43e01dfbc0 con 0x7f43e00a9ac0 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e01df020 con 0x7f43e00a9ac0 2024-03-29T10:46:12.588 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f43d4001b00 con 0x7f43e00a9ac0 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.603+0000 7f43dcff9640 10 monclient(hunting): dump: 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:12.589 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:12.590 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:12.590 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:12.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:12.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760366974 0 0) 0x7f43cc004eb0 con 0x7f43e00b4100 2024-03-29T10:46:12.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e01dfbc0 con 0x7f43e00b4100 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f43d8001d20 con 0x7f43e00b4100 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 10 monclient(hunting): dump: 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.591 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.592 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:12.592 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:12.592 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:12.592 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:12.592 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:12.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3817557083 0 0) 0x7f43e01df020 con 0x7f43e00a9ac0 2024-03-29T10:46:12.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.604+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 >> v1:172.21.15.94:6790/0 conn(0x7f43e013ee90 legacy=0x7f43e0140850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:12.594 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 >> v1:172.21.15.173:6789/0 conn(0x7f43e00b4100 legacy=0x7f43e0071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:12.594 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:12.594 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 1 monclient: found mon.a 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e013fd40 con 0x7f43e00a9ac0 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 5 monclient: authenticate success, global_id 4488 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 10 monclient: _renew_subs 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e00723a0 con 0x7f43e00a9ac0 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:12.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 10 monclient: _renew_subs 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e0141140 con 0x7f43e00a9ac0 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43e6493640 1 librados: init done 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43d4001170 con 0x7f43e00a9ac0 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.605+0000 7f43de7fc640 4 set_mon_vals no callback set 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.607+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f43d4004a90 con 0x7f43e00a9ac0 2024-03-29T10:46:12.596 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.607+0000 7f43dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.607+0000 7f43dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.607+0000 7f43dcff9640 10 monclient: dump: 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:12.597 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.607+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f43d401aa60 con 0x7f43e00a9ac0 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f43d4084950 con 0x7f43e00a9ac0 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43dcff9640 10 client.4488.objecter ms_dispatch 0x7f43e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43dcff9640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43dcff9640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43dcff9640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43e6493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43e6493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43e6493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c0005080 con 0x7f43e00a9ac0 2024-03-29T10:46:12.598 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.608+0000 7f43df7fe640 10 monclient: get_auth_request con 0x7f43cc06df10 auth_method 0 2024-03-29T10:46:12.599 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.609+0000 7f43dcff9640 10 client.4488.objecter ms_handle_connect 0x7f43cc06df10 2024-03-29T10:46:12.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.617+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43d40500e0 con 0x7f43e00a9ac0 2024-03-29T10:46:12.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.617+0000 7f43dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:12.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.617+0000 7f43dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:12.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.954+0000 7f43e6493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:12.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.954+0000 7f43e6493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:12.937 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.954+0000 7f43e6493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:12.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.954+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43c0005330 con 0x7f43e00a9ac0 2024-03-29T10:46:12.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.955+0000 7f43dcff9640 1 -- 172.21.15.94:0/3189788798 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f43d40521e0 con 0x7f43e00a9ac0 2024-03-29T10:46:12.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.955+0000 7f43dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:12.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.955+0000 7f43dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:12.939 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:12.939 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:12.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.959+0000 7f43e6493640 10 librados: watch_flush enter 2024-03-29T10:46:12.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.959+0000 7f43e6493640 10 librados: watch_flush exit 2024-03-29T10:46:12.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.959+0000 7f43e6493640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T10:46:12.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.959+0000 7f43e6493640 10 client.4488.objecter successfully canceled tick 2024-03-29T10:46:12.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.960+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 >> v1:172.21.15.94:6816/3829934808 conn(0x7f43cc06df10 legacy=0x7f43cc0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:12.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.960+0000 7f43e6493640 10 monclient: shutdown 2024-03-29T10:46:12.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.960+0000 7f43e6493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:12.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.960+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 >> v1:172.21.15.94:6789/0 conn(0x7f43e00a9ac0 legacy=0x7f43e01df4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:12.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.962+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 shutdown_connections 2024-03-29T10:46:12.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.962+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 >> 172.21.15.94:0/3189788798 conn(0x7f43e00b35e0 msgr2=0x7f43e00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:12.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.965+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 shutdown_connections 2024-03-29T10:46:12.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.965+0000 7f43e6493640 1 -- 172.21.15.94:0/3189788798 wait complete. 2024-03-29T10:46:12.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:12.965+0000 7f43e6493640 1 librados: shutdown 2024-03-29T10:46:12.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:19.771 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:19.938 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.956+0000 7f5fefd7c640 10 monclient: get_monmap_and_config 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.956+0000 7f5fefd7c640 10 monclient: build_initial_monmap 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.956+0000 7f5fefd7c640 10 monclient: monmap: 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:19.957168+0000 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:19.957168+0000 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:19.939 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:19.940 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.957+0000 7f5fefd7c640 1 Processor -- start 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.957+0000 7f5fefd7c640 1 -- start start 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.957+0000 7f5fefd7c640 10 monclient: init 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 10 monclient(hunting): picked mon.noname-a con 0x7f5fe81457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 10 monclient(hunting): picked mon.noname-c con 0x7f5fe813f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 10 monclient(hunting): picked mon.noname-b con 0x7f5fe80a9d70 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe80aaef0 con 0x7f5fe81457f0 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fedb18640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5fe81457f0 0x7f5fe813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:35842/0 (socket says 172.21.15.94:35842) 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fedb18640 1 -- 172.21.15.94:0/1458160052 learned_addr learned my addr 172.21.15.94:0/1458160052 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 1 -- 172.21.15.94:0/1458160052 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe8145bd0 con 0x7f5fe813f650 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 1 -- 172.21.15.94:0/1458160052 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe8145eb0 con 0x7f5fe80a9d70 2024-03-29T10:46:19.941 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:19.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.958+0000 7f5fefd7c640 10 monclient(hunting): authenticate will time out at 1231.226964s 2024-03-29T10:46:19.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377699337 0 0) 0x7f5fe8145bd0 con 0x7f5fe813f650 2024-03-29T10:46:19.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): _negotiate 2024-03-29T10:46:19.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:19.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:19.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:46:19.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd0003520 con 0x7f5fe813f650 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863646815 0 0) 0x7f5fe80aaef0 con 0x7f5fe81457f0 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): _negotiate 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe8145bd0 con 0x7f5fe81457f0 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681464164 0 0) 0x7f5fd0003520 con 0x7f5fe813f650 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 >> v1:172.21.15.94:6790/0 conn(0x7f5fe80a9d70 legacy=0x7f5fe814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 >> v1:172.21.15.94:6789/0 conn(0x7f5fe81457f0 legacy=0x7f5fe813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 10 monclient: _finish_hunting 0 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 1 monclient: found mon.noname-c 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.959+0000 7f5fecb16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:46:19.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe8146340 con 0x7f5fe813f650 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: _finish_auth 0 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: _check_auth_tickets 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5fdc001690 con 0x7f5fe813f650 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: dump: 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:19.946 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fefd7c640 5 monclient: authenticate success, global_id 4459 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fefd7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fdc0018b0 con 0x7f5fe813f650 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 1 -- 172.21.15.94:0/1458160052 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5fdc001c40 con 0x7f5fe813f650 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fecb16640 10 monclient: dump: 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:19.947 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fefd7c640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.960+0000 7f5fefd7c640 4 set_mon_vals no callback set 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.961+0000 7f5fefd7c640 10 monclient: shutdown 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.961+0000 7f5fefd7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.961+0000 7f5fefd7c640 1 -- 172.21.15.94:0/1458160052 >> v1:172.21.15.173:6789/0 conn(0x7f5fe813f650 legacy=0x7f5fe80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.962+0000 7f5fefd7c640 1 -- 172.21.15.94:0/1458160052 shutdown_connections 2024-03-29T10:46:19.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.962+0000 7f5fefd7c640 1 -- 172.21.15.94:0/1458160052 >> 172.21.15.94:0/1458160052 conn(0x7f5fe8066a00 msgr2=0x7f5fe8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:19.949 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.962+0000 7f5fefd7c640 1 -- 172.21.15.94:0/1458160052 shutdown_connections 2024-03-29T10:46:19.949 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.962+0000 7f5fefd7c640 1 -- 172.21.15.94:0/1458160052 wait complete. 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 10 monclient: build_initial_monmap 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 10 monclient: monmap: 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:19.953 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 1 librados: starting msgr at 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 1 librados: starting objecter 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 1 Processor -- start 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 1 -- start start 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 1 librados: setting wanted keys 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 1 librados: calling monclient init 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.963+0000 7f5fefd7c640 10 monclient: init 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:19.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 10 monclient(hunting): picked mon.noname-a con 0x7f5fe80a9d70 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 10 monclient(hunting): picked mon.noname-b con 0x7f5fe813f650 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 10 monclient(hunting): picked mon.noname-c con 0x7f5fe81457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe8142cf0 con 0x7f5fe80a9d70 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe8137b10 con 0x7f5fe813f650 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe80ac630 con 0x7f5fe81457f0 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fefd7c640 10 monclient(hunting): authenticate will time out at 1231.232793s 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fedb18640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5fe80a9d70 0x7f5fe813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:36416/0 (socket says 172.21.15.94:36416) 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fedb18640 1 -- 172.21.15.94:0/842191045 learned_addr learned my addr 172.21.15.94:0/842191045 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fe67fc640 10 client.?.objecter ms_handle_connect 0x7f5fe80a9d70 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.964+0000 7f5fe67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:19.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 client.?.objecter ms_handle_connect 0x7f5fe813f650 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 client.?.objecter ms_handle_connect 0x7f5fe81457f0 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118556942 0 0) 0x7f5fe8142cf0 con 0x7f5fe80a9d70 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd4004eb0 con 0x7f5fe80a9d70 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561188512 0 0) 0x7f5fe8146340 con 0x7f5fe813f650 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:46:19.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe8142cf0 con 0x7f5fe813f650 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 166126276 0 0) 0x7f5fe8142cf0 con 0x7f5fe813f650 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe8146340 con 0x7f5fe813f650 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 505575797 0 0) 0x7f5fd4004eb0 con 0x7f5fe80a9d70 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe8142cf0 con 0x7f5fe80a9d70 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5fdc001430 con 0x7f5fe813f650 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): dump: 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:19.957 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5fd8001890 con 0x7f5fe80a9d70 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.965+0000 7f5fe67fc640 10 monclient(hunting): dump: 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.958 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3206553721 0 0) 0x7f5fe8146340 con 0x7f5fe813f650 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 >> v1:172.21.15.94:6790/0 conn(0x7f5fe81457f0 legacy=0x7f5fe8137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 >> v1:172.21.15.94:6789/0 conn(0x7f5fe80a9d70 legacy=0x7f5fe813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 1 monclient: found mon.b 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe8137890 con 0x7f5fe813f650 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:19.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 5 monclient: authenticate success, global_id 4462 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 10 monclient: _renew_subs 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe81404b0 con 0x7f5fe813f650 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 10 monclient: _renew_subs 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe8137cd0 con 0x7f5fe813f650 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fefd7c640 1 librados: init done 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fdc001590 con 0x7f5fe813f650 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5fdc003a40 con 0x7f5fe813f650 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:19.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe67fc640 10 monclient: dump: 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.966+0000 7f5fe7fff640 4 set_mon_vals no callback set 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.967+0000 7f5fefd7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.967+0000 7f5fefd7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:19.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.967+0000 7f5fefd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.967+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc8005080 con 0x7f5fe813f650 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.968+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f5fdc019a10 con 0x7f5fe813f650 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.969+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f5fdc0836a0 con 0x7f5fe813f650 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.969+0000 7f5fe67fc640 10 client.4462.objecter ms_dispatch 0x7f5fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.969+0000 7f5fe67fc640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.969+0000 7f5fe67fc640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.969+0000 7f5fe67fc640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.969+0000 7f5fedb18640 10 monclient: get_auth_request con 0x7f5fd406def0 auth_method 0 2024-03-29T10:46:19.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.970+0000 7f5fe67fc640 10 client.4462.objecter ms_handle_connect 0x7f5fd406def0 2024-03-29T10:46:19.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.975+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fdc04ef80 con 0x7f5fe813f650 2024-03-29T10:46:19.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.975+0000 7f5fe67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:19.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:19.975+0000 7f5fe67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:20.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.385+0000 7f5fefd7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:20.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.385+0000 7f5fefd7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:20.367 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.385+0000 7f5fefd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:20.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.385+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fc8005330 con 0x7f5fe813f650 2024-03-29T10:46:20.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.386+0000 7f5fe67fc640 1 -- 172.21.15.94:0/842191045 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f5fdc050eb0 con 0x7f5fe813f650 2024-03-29T10:46:20.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.386+0000 7f5fe67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:20.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.386+0000 7f5fe67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:20.368 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:20.369 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:20.372 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 10 librados: watch_flush enter 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 10 librados: watch_flush exit 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 10 client.4462.objecter successfully canceled tick 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 >> v1:172.21.15.94:6816/3829934808 conn(0x7f5fd406def0 legacy=0x7f5fd40703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 10 monclient: shutdown 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 >> v1:172.21.15.173:6789/0 conn(0x7f5fe813f650 legacy=0x7f5fe8136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 shutdown_connections 2024-03-29T10:46:20.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.390+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 >> 172.21.15.94:0/842191045 conn(0x7f5fe8066a00 msgr2=0x7f5fe813ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:20.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.391+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 shutdown_connections 2024-03-29T10:46:20.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.391+0000 7f5fefd7c640 1 -- 172.21.15.94:0/842191045 wait complete. 2024-03-29T10:46:20.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:20.391+0000 7f5fefd7c640 1 librados: shutdown 2024-03-29T10:46:20.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:27.198 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:27.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.350+0000 7fb34a34d640 10 monclient: get_monmap_and_config 2024-03-29T10:46:27.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.350+0000 7fb34a34d640 10 monclient: build_initial_monmap 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.350+0000 7fb34a34d640 10 monclient: monmap: 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:27.351116+0000 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:27.351116+0000 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:27.333 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:27.334 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.351+0000 7fb34a34d640 1 Processor -- start 2024-03-29T10:46:27.334 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.351+0000 7fb34a34d640 1 -- start start 2024-03-29T10:46:27.334 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.351+0000 7fb34a34d640 10 monclient: init 2024-03-29T10:46:27.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:27.335 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3440a9750 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 10 monclient(hunting): picked mon.noname-a con 0x7fb34413e1e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 10 monclient(hunting): picked mon.noname-b con 0x7fb344071c30 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440ab300 con 0x7fb34413e1e0 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440ac2f0 con 0x7fb3440a9750 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3440ad8d0 con 0x7fb344071c30 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.352+0000 7fb34a34d640 10 monclient(hunting): authenticate will time out at 1238.620486s 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.353+0000 7fb3437fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb34413e1e0 0x7fb3440ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53588/0 (socket says 172.21.15.94:53588) 2024-03-29T10:46:27.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.353+0000 7fb3437fe640 1 -- 172.21.15.94:0/3351347518 learned_addr learned my addr 172.21.15.94:0/3351347518 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455900957 0 0) 0x7fb3440ab300 con 0x7fb34413e1e0 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb32c003520 con 0x7fb34413e1e0 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3828075579 0 0) 0x7fb32c003520 con 0x7fb34413e1e0 2024-03-29T10:46:27.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.354+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 >> v1:172.21.15.94:6790/0 conn(0x7fb344071c30 legacy=0x7fb3440ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:27.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.355+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 >> v1:172.21.15.173:6789/0 conn(0x7fb3440a9750 legacy=0x7fb34413dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:27.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:27.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 1 monclient: found mon.noname-a 2024-03-29T10:46:27.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:27.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:46:27.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3440ada60 con 0x7fb34413e1e0 2024-03-29T10:46:27.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb34a34d640 5 monclient: authenticate success, global_id 4497 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb34a34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb338001690 con 0x7fb34413e1e0 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.357+0000 7fb342ffd640 10 monclient: dump: 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.342 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb338001920 con 0x7fb34413e1e0 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb342ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb342ffd640 1 -- 172.21.15.94:0/3351347518 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb338001cb0 con 0x7fb34413e1e0 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb342ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:27.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb342ffd640 10 monclient: dump: 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 4 set_mon_vals no callback set 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 10 monclient: shutdown 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:27.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 1 -- 172.21.15.94:0/3351347518 >> v1:172.21.15.94:6789/0 conn(0x7fb34413e1e0 legacy=0x7fb3440ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 1 -- 172.21.15.94:0/3351347518 shutdown_connections 2024-03-29T10:46:27.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.358+0000 7fb34a34d640 1 -- 172.21.15.94:0/3351347518 >> 172.21.15.94:0/3351347518 conn(0x7fb34406b660 msgr2=0x7fb34406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:27.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.359+0000 7fb34a34d640 1 -- 172.21.15.94:0/3351347518 shutdown_connections 2024-03-29T10:46:27.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.359+0000 7fb34a34d640 1 -- 172.21.15.94:0/3351347518 wait complete. 2024-03-29T10:46:27.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.359+0000 7fb34a34d640 10 monclient: build_initial_monmap 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.359+0000 7fb34a34d640 10 monclient: monmap: 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.360+0000 7fb34a34d640 1 librados: starting msgr at 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.360+0000 7fb34a34d640 1 librados: starting objecter 2024-03-29T10:46:27.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.360+0000 7fb34a34d640 1 Processor -- start 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.360+0000 7fb34a34d640 1 -- start start 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.360+0000 7fb34a34d640 1 librados: setting wanted keys 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.360+0000 7fb34a34d640 1 librados: calling monclient init 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.360+0000 7fb34a34d640 10 monclient: init 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.361+0000 7fb34a34d640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.361+0000 7fb34a34d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 10 monclient(hunting): picked mon.noname-b con 0x7fb344071c30 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 10 monclient(hunting): picked mon.noname-c con 0x7fb3440a9750 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 10 monclient(hunting): picked mon.noname-a con 0x7fb34413e1e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb344140620 con 0x7fb34413e1e0 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb344140900 con 0x7fb344071c30 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3441395a0 con 0x7fb3440a9750 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb34a34d640 10 monclient(hunting): authenticate will time out at 1238.630248s 2024-03-29T10:46:27.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb3488ea640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fb34413e1e0 0x7fb3441e5ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:54218/0 (socket says 172.21.15.94:54218) 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb3488ea640 1 -- 172.21.15.94:0/2369734533 learned_addr learned my addr 172.21.15.94:0/2369734533 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 client.?.objecter ms_handle_connect 0x7fb34413e1e0 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747367440 0 0) 0x7fb344140620 con 0x7fb34413e1e0 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb330004eb0 con 0x7fb34413e1e0 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605823647 0 0) 0x7fb330004eb0 con 0x7fb34413e1e0 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb344140620 con 0x7fb34413e1e0 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb33c001c60 con 0x7fb34413e1e0 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:27.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): dump: 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.362+0000 7fb340ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.363+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3708671375 0 0) 0x7fb344140620 con 0x7fb34413e1e0 2024-03-29T10:46:27.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.363+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 >> v1:172.21.15.94:6790/0 conn(0x7fb3440a9750 legacy=0x7fb344140d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:27.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 >> v1:172.21.15.173:6789/0 conn(0x7fb344071c30 legacy=0x7fb3441ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:27.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 1 monclient: found mon.a 2024-03-29T10:46:27.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3441c9250 con 0x7fb34413e1e0 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 10 monclient: _renew_subs 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3441c9500 con 0x7fb34413e1e0 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 10 monclient: _renew_subs 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb344138b80 con 0x7fb34413e1e0 2024-03-29T10:46:27.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb34a34d640 1 librados: init done 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb33c0020c0 con 0x7fb34413e1e0 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fb33c003dc0 con 0x7fb34413e1e0 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb340ff9640 10 monclient: dump: 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:27.353 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.366+0000 7fb3427fc640 4 set_mon_vals no callback set 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.368+0000 7fb34a34d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.368+0000 7fb34a34d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.368+0000 7fb34a34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.368+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb324005080 con 0x7fb34413e1e0 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.368+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fb33c003f20 con 0x7fb34413e1e0 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.369+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fb33c084e40 con 0x7fb34413e1e0 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.369+0000 7fb340ff9640 10 client.4500.objecter ms_dispatch 0x7fb344000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.369+0000 7fb340ff9640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.369+0000 7fb340ff9640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.369+0000 7fb340ff9640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T10:46:27.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.369+0000 7fb343fff640 10 monclient: get_auth_request con 0x7fb33006dbf0 auth_method 0 2024-03-29T10:46:27.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.369+0000 7fb340ff9640 10 client.4500.objecter ms_handle_connect 0x7fb33006dbf0 2024-03-29T10:46:27.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.377+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb33c050720 con 0x7fb34413e1e0 2024-03-29T10:46:27.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.377+0000 7fb340ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:27.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.377+0000 7fb340ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:27.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.672+0000 7fb34a34d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:27.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.672+0000 7fb34a34d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:27.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.672+0000 7fb34a34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:27.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.672+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb324005330 con 0x7fb34413e1e0 2024-03-29T10:46:27.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.673+0000 7fb340ff9640 1 -- 172.21.15.94:0/2369734533 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fb33c052650 con 0x7fb34413e1e0 2024-03-29T10:46:27.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.673+0000 7fb340ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:27.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.673+0000 7fb340ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:27.656 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:27.656 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:27.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 10 librados: watch_flush enter 2024-03-29T10:46:27.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 10 librados: watch_flush exit 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 10 client.4500.objecter successfully canceled tick 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 >> v1:172.21.15.94:6816/3829934808 conn(0x7fb33006dbf0 legacy=0x7fb3300700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 10 monclient: shutdown 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 >> v1:172.21.15.94:6789/0 conn(0x7fb34413e1e0 legacy=0x7fb3441e5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 shutdown_connections 2024-03-29T10:46:27.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.677+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 >> 172.21.15.94:0/2369734533 conn(0x7fb34406b660 msgr2=0x7fb34406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:27.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.678+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 shutdown_connections 2024-03-29T10:46:27.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.678+0000 7fb34a34d640 1 -- 172.21.15.94:0/2369734533 wait complete. 2024-03-29T10:46:27.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:27.678+0000 7fb34a34d640 1 librados: shutdown 2024-03-29T10:46:27.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:34.485 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:34.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.639+0000 7f74f84d6640 10 monclient: get_monmap_and_config 2024-03-29T10:46:34.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.639+0000 7f74f84d6640 10 monclient: build_initial_monmap 2024-03-29T10:46:34.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.639+0000 7f74f84d6640 10 monclient: monmap: 2024-03-29T10:46:34.623 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:34.641527+0000 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:34.641527+0000 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:34.624 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:34.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.640+0000 7f74f84d6640 1 Processor -- start 2024-03-29T10:46:34.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.640+0000 7f74f84d6640 1 -- start start 2024-03-29T10:46:34.625 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.640+0000 7f74f84d6640 10 monclient: init 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 10 monclient(hunting): picked mon.noname-c con 0x7f74f0074650 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 10 monclient(hunting): picked mon.noname-a con 0x7f74f0072e80 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 10 monclient(hunting): picked mon.noname-b con 0x7f74f006f510 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f006f080 con 0x7f74f0072e80 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f0070900 con 0x7f74f0074650 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f0143d10 con 0x7f74f006f510 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:34.626 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.641+0000 7f74f84d6640 10 monclient(hunting): authenticate will time out at 1245.910789s 2024-03-29T10:46:34.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5a71640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f74f0072e80 0x7f74f01499e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57126/0 (socket says 172.21.15.94:57126) 2024-03-29T10:46:34.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5a71640 1 -- 172.21.15.94:0/1897808847 learned_addr learned my addr 172.21.15.94:0/1897808847 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:34.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166500172 0 0) 0x7f74f006f080 con 0x7f74f0072e80 2024-03-29T10:46:34.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5270640 10 monclient(hunting): _negotiate 2024-03-29T10:46:34.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5270640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:34.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5270640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:34.631 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5270640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.643+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc003520 con 0x7f74f0072e80 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.644+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 675207862 0 0) 0x7f74dc003520 con 0x7f74f0072e80 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.644+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 >> v1:172.21.15.94:6790/0 conn(0x7f74f006f510 legacy=0x7f74f0151d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.644+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 >> v1:172.21.15.173:6789/0 conn(0x7f74f0074650 legacy=0x7f74f0143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: _finish_hunting 0 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 1 monclient: found mon.noname-a 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:46:34.632 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f006e8d0 con 0x7f74f0072e80 2024-03-29T10:46:34.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: _finish_auth 0 2024-03-29T10:46:34.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: _check_auth_tickets 2024-03-29T10:46:34.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:34.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74ec001690 con 0x7f74f0072e80 2024-03-29T10:46:34.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:34.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:34.633 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: dump: 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:34.634 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f84d6640 5 monclient: authenticate success, global_id 4503 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f84d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74ec001860 con 0x7f74f0072e80 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 1 -- 172.21.15.94:0/1897808847 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74ec001cb0 con 0x7f74f0072e80 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f5270640 10 monclient: dump: 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.635 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f84d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.645+0000 7f74f84d6640 4 set_mon_vals no callback set 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.646+0000 7f74f84d6640 10 monclient: shutdown 2024-03-29T10:46:34.636 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.646+0000 7f74f84d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:34.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.646+0000 7f74f84d6640 1 -- 172.21.15.94:0/1897808847 >> v1:172.21.15.94:6789/0 conn(0x7f74f0072e80 legacy=0x7f74f01499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.646+0000 7f74f84d6640 1 -- 172.21.15.94:0/1897808847 shutdown_connections 2024-03-29T10:46:34.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.646+0000 7f74f84d6640 1 -- 172.21.15.94:0/1897808847 >> 172.21.15.94:0/1897808847 conn(0x7f74f00b4200 msgr2=0x7f74f00b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:34.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.647+0000 7f74f84d6640 1 -- 172.21.15.94:0/1897808847 shutdown_connections 2024-03-29T10:46:34.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.647+0000 7f74f84d6640 1 -- 172.21.15.94:0/1897808847 wait complete. 2024-03-29T10:46:34.648 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.648+0000 7f74f84d6640 10 monclient: build_initial_monmap 2024-03-29T10:46:34.648 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.648+0000 7f74f84d6640 10 monclient: monmap: 2024-03-29T10:46:34.648 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:34.648 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:34.648 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.649+0000 7f74f84d6640 1 librados: starting msgr at 2024-03-29T10:46:34.649 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.649+0000 7f74f84d6640 1 librados: starting objecter 2024-03-29T10:46:34.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.649+0000 7f74f84d6640 1 Processor -- start 2024-03-29T10:46:34.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.649+0000 7f74f84d6640 1 -- start start 2024-03-29T10:46:34.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.649+0000 7f74f84d6640 1 librados: setting wanted keys 2024-03-29T10:46:34.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.649+0000 7f74f84d6640 1 librados: calling monclient init 2024-03-29T10:46:34.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.649+0000 7f74f84d6640 10 monclient: init 2024-03-29T10:46:34.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.650+0000 7f74f84d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:34.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.650+0000 7f74f84d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:34.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.650+0000 7f74f84d6640 10 monclient(hunting): picked mon.noname-c con 0x7f74f006f510 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:34.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.650+0000 7f74f84d6640 10 monclient(hunting): picked mon.noname-a con 0x7f74f0074650 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:34.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.651+0000 7f74f84d6640 10 monclient(hunting): picked mon.noname-b con 0x7f74f01cfd70 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:34.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.651+0000 7f74f84d6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f01cf8e0 con 0x7f74f0074650 2024-03-29T10:46:34.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.651+0000 7f74f84d6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f01403e0 con 0x7f74f01cfd70 2024-03-29T10:46:34.651 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.651+0000 7f74f84d6640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f01d2d60 con 0x7f74f006f510 2024-03-29T10:46:34.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.651+0000 7f74f84d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:34.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.651+0000 7f74f84d6640 10 monclient(hunting): authenticate will time out at 1245.920384s 2024-03-29T10:46:34.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.652+0000 7f74f5a71640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f74f0074650 0x7f74f013f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:57528/0 (socket says 172.21.15.94:57528) 2024-03-29T10:46:34.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.652+0000 7f74f5a71640 1 -- 172.21.15.94:0/1650099782 learned_addr learned my addr 172.21.15.94:0/1650099782 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:34.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.653+0000 7f74e6ffd640 10 client.?.objecter ms_handle_connect 0x7f74f0074650 2024-03-29T10:46:34.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.653+0000 7f74e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:34.652 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.653+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021315863 0 0) 0x7f74f01cf8e0 con 0x7f74f0074650 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.653+0000 7f74e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d4004eb0 con 0x7f74f0074650 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 client.?.objecter ms_handle_connect 0x7f74f01cfd70 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 client.?.objecter ms_handle_connect 0x7f74f006f510 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788585666 0 0) 0x7f74f01403e0 con 0x7f74f01cfd70 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:34.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.654+0000 7f74e6ffd640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f01cf8e0 con 0x7f74f01cfd70 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896044013 0 0) 0x7f74f01d2d60 con 0x7f74f006f510 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f01403e0 con 0x7f74f006f510 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2440770770 0 0) 0x7f74d4004eb0 con 0x7f74f0074650 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74f01d2d60 con 0x7f74f0074650 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74ec001b70 con 0x7f74f0074650 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:34.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:34.655 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.655+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1173704888 0 0) 0x7f74f01d2d60 con 0x7f74f0074650 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 >> v1:172.21.15.94:6790/0 conn(0x7f74f006f510 legacy=0x7f74f01d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 >> v1:172.21.15.173:6789/0 conn(0x7f74f01cfd70 legacy=0x7f74f013fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 1 monclient: found mon.a 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f01d10f0 con 0x7f74f0074650 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74ec001d70 con 0x7f74f0074650 2024-03-29T10:46:34.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74ec004dd0 con 0x7f74f0074650 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.656+0000 7f74e6ffd640 10 monclient: dump: 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:34.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 5 monclient: authenticate success, global_id 4506 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f4a6f640 4 set_mon_vals no callback set 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 10 monclient: _renew_subs 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f0140660 con 0x7f74f0074650 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 10 monclient: _renew_subs 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f0146a20 con 0x7f74f0074650 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.657+0000 7f74f84d6640 1 librados: init done 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.659+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f74ec01ada0 con 0x7f74f0074650 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.660+0000 7f74f84d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.660+0000 7f74f84d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:34.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.660+0000 7f74f84d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.660+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f0067bf0 con 0x7f74f0074650 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.660+0000 7f74f6272640 10 monclient: get_auth_request con 0x7f74d406dcc0 auth_method 0 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.661+0000 7f74e6ffd640 10 client.4506.objecter ms_handle_connect 0x7f74d406dcc0 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.661+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f74ec084b90 con 0x7f74f0074650 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.661+0000 7f74e6ffd640 10 client.4506.objecter ms_dispatch 0x7f74f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.661+0000 7f74e6ffd640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.661+0000 7f74e6ffd640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:34.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.661+0000 7f74e6ffd640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T10:46:34.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.668+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74ec050470 con 0x7f74f0074650 2024-03-29T10:46:34.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.668+0000 7f74e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:34.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.668+0000 7f74e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:34.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.978+0000 7f74f84d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:34.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.978+0000 7f74f84d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:34.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.978+0000 7f74f84d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:34.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.978+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74f0075f10 con 0x7f74f0074650 2024-03-29T10:46:34.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.981+0000 7f74e6ffd640 1 -- 172.21.15.94:0/1650099782 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f74ec0523a0 con 0x7f74f0074650 2024-03-29T10:46:34.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.981+0000 7f74e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:34.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.981+0000 7f74e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:34.965 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:34.965 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:34.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 10 librados: watch_flush enter 2024-03-29T10:46:34.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 10 librados: watch_flush exit 2024-03-29T10:46:34.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T10:46:34.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 10 client.4506.objecter successfully canceled tick 2024-03-29T10:46:34.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 >> v1:172.21.15.94:6816/3829934808 conn(0x7f74d406dcc0 legacy=0x7f74d4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 10 monclient: shutdown 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 >> v1:172.21.15.94:6789/0 conn(0x7f74f0074650 legacy=0x7f74f013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 shutdown_connections 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.984+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 >> 172.21.15.94:0/1650099782 conn(0x7f74f00b4200 msgr2=0x7f74f00af170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.985+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 shutdown_connections 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.985+0000 7f74f84d6640 1 -- 172.21.15.94:0/1650099782 wait complete. 2024-03-29T10:46:34.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:34.985+0000 7f74f84d6640 1 librados: shutdown 2024-03-29T10:46:34.983 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:41.791 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.955+0000 7ff93820f640 10 monclient: get_monmap_and_config 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.955+0000 7ff93820f640 10 monclient: build_initial_monmap 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.955+0000 7ff93820f640 10 monclient: monmap: 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:41.957474+0000 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:41.957474+0000 2024-03-29T10:46:41.939 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:41.940 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:41.940 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:41.940 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:41.940 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:41.940 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:41.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.956+0000 7ff93820f640 1 Processor -- start 2024-03-29T10:46:41.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.956+0000 7ff93820f640 1 -- start start 2024-03-29T10:46:41.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.956+0000 7ff93820f640 10 monclient: init 2024-03-29T10:46:41.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.957+0000 7ff93820f640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:41.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.957+0000 7ff93820f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:41.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 10 monclient(hunting): picked mon.noname-a con 0x7ff9301457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 10 monclient(hunting): picked mon.noname-b con 0x7ff93013f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9300a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9300aaef0 con 0x7ff9301457f0 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff930145bd0 con 0x7ff9300a9d70 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff930145eb0 con 0x7ff93013f650 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff93820f640 10 monclient(hunting): authenticate will time out at 1253.227325s 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff935fab640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff9301457f0 0x7ff93013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41390/0 (socket says 172.21.15.94:41390) 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff935fab640 1 -- 172.21.15.94:0/778534741 learned_addr learned my addr 172.21.15.94:0/778534741 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:41.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447955540 0 0) 0x7ff9300aaef0 con 0x7ff9301457f0 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): my global_id is 4509 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c003520 con 0x7ff9301457f0 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581297295 0 0) 0x7ff930145eb0 con 0x7ff93013f650 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:46:41.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.958+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9300aaef0 con 0x7ff93013f650 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1154864896 0 0) 0x7ff91c003520 con 0x7ff9301457f0 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 >> v1:172.21.15.94:6790/0 conn(0x7ff93013f650 legacy=0x7ff9300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 >> v1:172.21.15.173:6789/0 conn(0x7ff9300a9d70 legacy=0x7ff93014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 monclient: found mon.noname-a 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff930146340 con 0x7ff9301457f0 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: _finish_auth 0 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:41.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff92c001690 con 0x7ff9301457f0 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: dump: 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:41.946 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:41.947 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:41.947 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:41.947 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:41.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff93820f640 5 monclient: authenticate success, global_id 4509 2024-03-29T10:46:41.947 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff93820f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:41.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff92c0017f0 con 0x7ff9301457f0 2024-03-29T10:46:41.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:41.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 1 -- 172.21.15.94:0/778534741 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff92c001c00 con 0x7ff9301457f0 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff93820f640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff93820f640 4 set_mon_vals no callback set 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.959+0000 7ff934fa9640 10 monclient: dump: 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:41.954 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.960+0000 7ff93820f640 10 monclient: shutdown 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.960+0000 7ff93820f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.960+0000 7ff93820f640 1 -- 172.21.15.94:0/778534741 >> v1:172.21.15.94:6789/0 conn(0x7ff9301457f0 legacy=0x7ff93013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.960+0000 7ff93820f640 1 -- 172.21.15.94:0/778534741 shutdown_connections 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.960+0000 7ff93820f640 1 -- 172.21.15.94:0/778534741 >> 172.21.15.94:0/778534741 conn(0x7ff930066a00 msgr2=0x7ff930066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.960+0000 7ff93820f640 1 -- 172.21.15.94:0/778534741 shutdown_connections 2024-03-29T10:46:41.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.961+0000 7ff93820f640 1 -- 172.21.15.94:0/778534741 wait complete. 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.961+0000 7ff93820f640 10 monclient: build_initial_monmap 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.961+0000 7ff93820f640 10 monclient: monmap: 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:41.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.962+0000 7ff93820f640 1 librados: starting msgr at 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.962+0000 7ff93820f640 1 librados: starting objecter 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.962+0000 7ff93820f640 1 Processor -- start 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.962+0000 7ff93820f640 1 -- start start 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.962+0000 7ff93820f640 1 librados: setting wanted keys 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.962+0000 7ff93820f640 1 librados: calling monclient init 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.962+0000 7ff93820f640 10 monclient: init 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 10 monclient(hunting): picked mon.noname-b con 0x7ff9300a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 10 monclient(hunting): picked mon.noname-a con 0x7ff93013f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:41.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9301457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9301db2d0 con 0x7ff93013f650 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9301387a0 con 0x7ff9300a9d70 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9300b17c0 con 0x7ff9301457f0 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff93820f640 10 monclient(hunting): authenticate will time out at 1253.232584s 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff9357aa640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff93013f650 0x7ff930137980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:41588/0 (socket says 172.21.15.94:41588) 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff9357aa640 1 -- 172.21.15.94:0/2796283781 learned_addr learned my addr 172.21.15.94:0/2796283781 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff9227fc640 10 client.?.objecter ms_handle_connect 0x7ff93013f650 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff9227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff9227fc640 10 client.?.objecter ms_handle_connect 0x7ff9301457f0 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff9227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:41.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.963+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182275711 0 0) 0x7ff9301db2d0 con 0x7ff93013f650 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff914004eb0 con 0x7ff93013f650 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841220182 0 0) 0x7ff9300b17c0 con 0x7ff9301457f0 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9301db2d0 con 0x7ff9301457f0 2024-03-29T10:46:41.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1235661211 0 0) 0x7ff914004eb0 con 0x7ff93013f650 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9300b17c0 con 0x7ff93013f650 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff924001a10 con 0x7ff93013f650 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): dump: 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:41.960 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030538193 0 0) 0x7ff9301db2d0 con 0x7ff9301457f0 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff914004eb0 con 0x7ff9301457f0 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff928001bb0 con 0x7ff9301457f0 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient(hunting): dump: 2024-03-29T10:46:41.961 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2439050217 0 0) 0x7ff914004eb0 con 0x7ff9301457f0 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 >> v1:172.21.15.173:6789/0 conn(0x7ff9300a9d70 legacy=0x7ff9300ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:41.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 >> v1:172.21.15.94:6789/0 conn(0x7ff93013f650 legacy=0x7ff930137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 monclient: found mon.c 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9301db760 con 0x7ff9301457f0 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff9227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff93820f640 5 monclient: authenticate success, global_id 4523 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff93820f640 10 monclient: _renew_subs 2024-03-29T10:46:41.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff93820f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.964+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9300abd10 con 0x7ff9301457f0 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff93820f640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff93820f640 10 monclient: _renew_subs 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff93820f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9300b0f00 con 0x7ff9301457f0 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff93820f640 1 librados: init done 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff928002060 con 0x7ff9301457f0 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff9227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff928003b30 con 0x7ff9301457f0 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff9227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff9227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:41.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff9237fe640 4 set_mon_vals no callback set 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.965+0000 7ff9227fc640 10 monclient: dump: 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.966+0000 7ff93820f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:41.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.966+0000 7ff93820f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.966+0000 7ff93820f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.966+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff918005080 con 0x7ff9301457f0 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.967+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7ff928003c90 con 0x7ff9301457f0 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.968+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7ff928083b30 con 0x7ff9301457f0 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.968+0000 7ff9227fc640 10 client.4523.objecter ms_dispatch 0x7ff930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.968+0000 7ff9227fc640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.968+0000 7ff9227fc640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.968+0000 7ff9227fc640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.968+0000 7ff935fab640 10 monclient: get_auth_request con 0x7ff91406df40 auth_method 0 2024-03-29T10:46:41.966 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.968+0000 7ff9227fc640 10 client.4523.objecter ms_handle_connect 0x7ff91406df40 2024-03-29T10:46:41.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.976+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff92804f470 con 0x7ff9301457f0 2024-03-29T10:46:41.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.976+0000 7ff9227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:41.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:41.976+0000 7ff9227fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:42.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.344+0000 7ff93820f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:42.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.344+0000 7ff93820f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:42.327 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.344+0000 7ff93820f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:46:42.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.344+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff918005330 con 0x7ff9301457f0 2024-03-29T10:46:42.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.345+0000 7ff9227fc640 1 -- 172.21.15.94:0/2796283781 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7ff9280831e0 con 0x7ff9301457f0 2024-03-29T10:46:42.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.345+0000 7ff9227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:42.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.345+0000 7ff9227fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:42.329 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:42.329 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.348+0000 7ff93820f640 10 librados: watch_flush enter 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.348+0000 7ff93820f640 10 librados: watch_flush exit 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.348+0000 7ff93820f640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.348+0000 7ff93820f640 10 client.4523.objecter successfully canceled tick 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.348+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 >> v1:172.21.15.94:6816/3829934808 conn(0x7ff91406df40 legacy=0x7ff914070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 10 monclient: shutdown 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 >> v1:172.21.15.94:6790/0 conn(0x7ff9301457f0 legacy=0x7ff930138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 shutdown_connections 2024-03-29T10:46:42.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 >> 172.21.15.94:0/2796283781 conn(0x7ff930066a00 msgr2=0x7ff9300b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:42.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 shutdown_connections 2024-03-29T10:46:42.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 1 -- 172.21.15.94:0/2796283781 wait complete. 2024-03-29T10:46:42.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:42.349+0000 7ff93820f640 1 librados: shutdown 2024-03-29T10:46:42.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:49.152 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.299+0000 7faff6954640 10 monclient: get_monmap_and_config 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.299+0000 7faff6954640 10 monclient: build_initial_monmap 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.299+0000 7faff6954640 10 monclient: monmap: 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:49.300705+0000 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:49.300705+0000 2024-03-29T10:46:49.282 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:49.283 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:49.283 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:49.283 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:49.283 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:49.283 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.300+0000 7faff6954640 1 Processor -- start 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 1 -- start start 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient: init 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient(hunting): picked mon.noname-a con 0x7faff01457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient(hunting): picked mon.noname-b con 0x7faff013f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient(hunting): picked mon.noname-c con 0x7faff00a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff00aaef0 con 0x7faff01457f0 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff0145bd0 con 0x7faff00a9d70 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff0145eb0 con 0x7faff013f650 2024-03-29T10:46:49.285 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.301+0000 7faff6954640 10 monclient(hunting): authenticate will time out at 1260.569968s 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafef7fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7faff013f650 0x7faff00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:55484/0 (socket says 172.21.15.94:55484) 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafef7fe640 1 -- 172.21.15.94:0/3947926209 learned_addr learned my addr 172.21.15.94:0/3947926209 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422679324 0 0) 0x7faff0145eb0 con 0x7faff013f650 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd8003520 con 0x7faff013f650 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254428002 0 0) 0x7faff0145bd0 con 0x7faff00a9d70 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:49.286 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:46:49.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff0145eb0 con 0x7faff00a9d70 2024-03-29T10:46:49.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263130467 0 0) 0x7fafd8003520 con 0x7faff013f650 2024-03-29T10:46:49.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 >> v1:172.21.15.173:6789/0 conn(0x7faff00a9d70 legacy=0x7faff014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.287 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.302+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 >> v1:172.21.15.94:6789/0 conn(0x7faff01457f0 legacy=0x7faff013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:49.290 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.303+0000 7fafeeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 1 monclient: found mon.noname-b 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0146340 con 0x7faff013f650 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fafe4001690 con 0x7faff013f650 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: dump: 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:49.291 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7faff6954640 5 monclient: authenticate success, global_id 4526 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7faff6954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe40018b0 con 0x7faff013f650 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 1 -- 172.21.15.94:0/3947926209 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fafe4001c40 con 0x7faff013f650 2024-03-29T10:46:49.292 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7fafeeffd640 10 monclient: dump: 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7faff6954640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.304+0000 7faff6954640 4 set_mon_vals no callback set 2024-03-29T10:46:49.293 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.305+0000 7faff6954640 10 monclient: shutdown 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.305+0000 7faff6954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.305+0000 7faff6954640 1 -- 172.21.15.94:0/3947926209 >> v1:172.21.15.94:6790/0 conn(0x7faff013f650 legacy=0x7faff00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.305+0000 7faff6954640 1 -- 172.21.15.94:0/3947926209 shutdown_connections 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.305+0000 7faff6954640 1 -- 172.21.15.94:0/3947926209 >> 172.21.15.94:0/3947926209 conn(0x7faff0066a00 msgr2=0x7faff0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.305+0000 7faff6954640 1 -- 172.21.15.94:0/3947926209 shutdown_connections 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.305+0000 7faff6954640 1 -- 172.21.15.94:0/3947926209 wait complete. 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.306+0000 7faff6954640 10 monclient: build_initial_monmap 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.306+0000 7faff6954640 10 monclient: monmap: 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:49.294 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.307+0000 7faff6954640 1 librados: starting msgr at 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.307+0000 7faff6954640 1 librados: starting objecter 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.307+0000 7faff6954640 1 Processor -- start 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.307+0000 7faff6954640 1 -- start start 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.307+0000 7faff6954640 1 librados: setting wanted keys 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.307+0000 7faff6954640 1 librados: calling monclient init 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.307+0000 7faff6954640 10 monclient: init 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.308+0000 7faff6954640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:49.295 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.308+0000 7faff6954640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:49.296 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.308+0000 7faff6954640 10 monclient(hunting): picked mon.noname-b con 0x7faff00a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:49.296 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.308+0000 7faff6954640 10 monclient(hunting): picked mon.noname-c con 0x7faff013f650 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:49.296 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.308+0000 7fafeffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7faff00a9d70 0x7faff0142710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:42390/0 (socket says 172.21.15.94:42390) 2024-03-29T10:46:49.296 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.308+0000 7fafeffff640 1 -- 172.21.15.94:0/1856336058 learned_addr learned my addr 172.21.15.94:0/1856336058 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.308+0000 7faff6954640 10 monclient(hunting): picked mon.noname-a con 0x7faff01457f0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7fafecff9640 10 client.?.objecter ms_handle_connect 0x7faff00a9d70 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7fafecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff013bb50 con 0x7faff01457f0 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff01d7e00 con 0x7faff00a9d70 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff00b0d60 con 0x7faff013f650 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7faff6954640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7faff6954640 10 monclient(hunting): authenticate will time out at 1260.577464s 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7fafecff9640 10 client.?.objecter ms_handle_connect 0x7faff013f650 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.309+0000 7fafecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673582479 0 0) 0x7faff01d7e00 con 0x7faff00a9d70 2024-03-29T10:46:49.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafdc004eb0 con 0x7faff00a9d70 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125926813 0 0) 0x7faff00b0d60 con 0x7faff013f650 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): _negotiate 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff01d7e00 con 0x7faff013f650 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1555897874 0 0) 0x7fafdc004eb0 con 0x7faff00a9d70 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faff00b0d60 con 0x7faff00a9d70 2024-03-29T10:46:49.302 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fafe00017d0 con 0x7faff00a9d70 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): dump: 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:49.303 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834811268 0 0) 0x7faff01d7e00 con 0x7faff013f650 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafdc004eb0 con 0x7faff013f650 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fafe4001610 con 0x7faff013f650 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 10 monclient(hunting): dump: 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:49.304 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3366759492 0 0) 0x7faff00b0d60 con 0x7faff00a9d70 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 >> v1:172.21.15.94:6790/0 conn(0x7faff013f650 legacy=0x7faff01d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.310+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 >> v1:172.21.15.94:6789/0 conn(0x7faff01457f0 legacy=0x7faff00b0650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 10 monclient: _finish_hunting 0 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 1 monclient: found mon.b 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff00b0ef0 con 0x7faff00a9d70 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 10 monclient: _finish_auth 0 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 10 monclient: _check_auth_tickets 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7fafecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:49.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7faff6954640 5 monclient: authenticate success, global_id 4471 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7faff6954640 10 monclient: _renew_subs 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7faff6954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.311+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff013fda0 con 0x7faff00a9d70 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7faff6954640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7faff6954640 10 monclient: _renew_subs 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7faff6954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faff01402d0 con 0x7faff00a9d70 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7faff6954640 1 librados: init done 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafe0001ce0 con 0x7faff00a9d70 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7fafecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fafe0004a00 con 0x7faff00a9d70 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7fafecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7fafecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:49.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7fafee7fc640 4 set_mon_vals no callback set 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.312+0000 7fafecff9640 10 monclient: dump: 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.314+0000 7faff6954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.314+0000 7faff6954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:49.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.314+0000 7faff6954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.314+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd0005080 con 0x7faff00a9d70 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.314+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fafe0004b60 con 0x7faff00a9d70 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.315+0000 7fafef7fe640 10 monclient: get_auth_request con 0x7fafdc06de80 auth_method 0 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.315+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fafe0084a10 con 0x7faff00a9d70 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.315+0000 7fafecff9640 10 client.4471.objecter ms_dispatch 0x7faff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.315+0000 7fafecff9640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.315+0000 7fafecff9640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.315+0000 7fafecff9640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-29T10:46:49.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.315+0000 7fafecff9640 10 client.4471.objecter ms_handle_connect 0x7fafdc06de80 2024-03-29T10:46:49.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.323+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafe00502f0 con 0x7faff00a9d70 2024-03-29T10:46:49.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.323+0000 7fafecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:49.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.323+0000 7fafecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:49.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.633+0000 7faff6954640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:49.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.633+0000 7faff6954640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:49.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.633+0000 7faff6954640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:46:49.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.633+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafd0005330 con 0x7faff00a9d70 2024-03-29T10:46:49.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.634+0000 7fafecff9640 1 -- 172.21.15.94:0/1856336058 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fafe0052220 con 0x7faff00a9d70 2024-03-29T10:46:49.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.634+0000 7fafecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:49.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.634+0000 7fafecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:49.617 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:49.617 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.637+0000 7faff6954640 10 librados: watch_flush enter 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 10 librados: watch_flush exit 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 10 client.4471.objecter successfully canceled tick 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 >> v1:172.21.15.94:6816/3829934808 conn(0x7fafdc06de80 legacy=0x7fafdc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 10 monclient: shutdown 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:49.620 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 >> v1:172.21.15.173:6789/0 conn(0x7faff00a9d70 legacy=0x7faff0142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:49.621 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 shutdown_connections 2024-03-29T10:46:49.621 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 >> 172.21.15.94:0/1856336058 conn(0x7faff0066a00 msgr2=0x7faff00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:49.621 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 shutdown_connections 2024-03-29T10:46:49.621 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 1 -- 172.21.15.94:0/1856336058 wait complete. 2024-03-29T10:46:49.621 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:49.638+0000 7faff6954640 1 librados: shutdown 2024-03-29T10:46:49.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:56.443 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.606+0000 7ff6539ba640 10 monclient: get_monmap_and_config 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.606+0000 7ff6539ba640 10 monclient: build_initial_monmap 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.606+0000 7ff6539ba640 10 monclient: monmap: 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:46:56.607637+0000 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:46:56.607637+0000 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:46:56.589 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:46:56.590 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:56.590 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.608+0000 7ff6539ba640 1 Processor -- start 2024-03-29T10:46:56.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.608+0000 7ff6539ba640 1 -- start start 2024-03-29T10:46:56.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.608+0000 7ff6539ba640 10 monclient: init 2024-03-29T10:46:56.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:56.592 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c1422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c0b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c142c80 con 0x7ff64c1422b0 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0aae40 con 0x7ff64c0b17d0 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c13fbd0 con 0x7ff64c142690 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:56.593 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.609+0000 7ff6539ba640 10 monclient(hunting): authenticate will time out at 1267.877409s 2024-03-29T10:46:56.594 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.611+0000 7ff650f55640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff64c142690 0x7ff64c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:42182/0 (socket says 172.21.15.94:42182) 2024-03-29T10:46:56.594 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.611+0000 7ff650f55640 1 -- 172.21.15.94:0/1704363246 learned_addr learned my addr 172.21.15.94:0/1704363246 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:56.594 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397970021 0 0) 0x7ff64c13fbd0 con 0x7ff64c142690 2024-03-29T10:46:56.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.595 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638003520 con 0x7ff64c142690 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2774660185 0 0) 0x7ff638003520 con 0x7ff64c142690 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.612+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 >> v1:172.21.15.173:6789/0 conn(0x7ff64c0b17d0 legacy=0x7ff64c0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 >> v1:172.21.15.94:6789/0 conn(0x7ff64c1422b0 legacy=0x7ff64c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:56.600 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 1 monclient: found mon.noname-b 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c151940 con 0x7ff64c142690 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff6539ba640 5 monclient: authenticate success, global_id 4532 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff6539ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff648001a10 con 0x7ff64c142690 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: dump: 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:56.601 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff6539ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff648001b70 con 0x7ff64c142690 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.614+0000 7ff643fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff6539ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff6539ba640 4 set_mon_vals no callback set 2024-03-29T10:46:56.602 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff643fff640 1 -- 172.21.15.94:0/1704363246 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff648001fc0 con 0x7ff64c142690 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff643fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff643fff640 10 monclient: dump: 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff6539ba640 10 monclient: shutdown 2024-03-29T10:46:56.603 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff6539ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:56.604 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.615+0000 7ff6539ba640 1 -- 172.21.15.94:0/1704363246 >> v1:172.21.15.94:6790/0 conn(0x7ff64c142690 legacy=0x7ff64c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.604 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.618+0000 7ff6539ba640 1 -- 172.21.15.94:0/1704363246 shutdown_connections 2024-03-29T10:46:56.604 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.618+0000 7ff6539ba640 1 -- 172.21.15.94:0/1704363246 >> 172.21.15.94:0/1704363246 conn(0x7ff64c066a00 msgr2=0x7ff64c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.619+0000 7ff6539ba640 1 -- 172.21.15.94:0/1704363246 shutdown_connections 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.619+0000 7ff6539ba640 1 -- 172.21.15.94:0/1704363246 wait complete. 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 10 monclient: build_initial_monmap 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 10 monclient: monmap: 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:46:56.605 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 1 librados: starting msgr at 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 1 librados: starting objecter 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 1 Processor -- start 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 1 -- start start 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 1 librados: setting wanted keys 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 1 librados: calling monclient init 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.620+0000 7ff6539ba640 10 monclient: init 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:56.606 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c0b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff64c1422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff64c142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c071ee0 con 0x7ff64c1422b0 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0ac9f0 con 0x7ff64c0b17d0 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c0accd0 con 0x7ff64c142690 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:56.607 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.621+0000 7ff6539ba640 10 monclient(hunting): authenticate will time out at 1267.889859s 2024-03-29T10:46:56.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.625+0000 7ff650f55640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff64c1422b0 0x7ff64c06f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:53701/0 (socket says 172.21.15.94:53701) 2024-03-29T10:46:56.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.625+0000 7ff650f55640 1 -- 172.21.15.94:0/2833699455 learned_addr learned my addr 172.21.15.94:0/2833699455 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:46:56.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 client.?.objecter ms_handle_connect 0x7ff64c1422b0 2024-03-29T10:46:56.608 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:56.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 client.?.objecter ms_handle_connect 0x7ff64c142690 2024-03-29T10:46:56.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:56.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786045359 0 0) 0x7ff64c071ee0 con 0x7ff64c1422b0 2024-03-29T10:46:56.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.609 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff630004eb0 con 0x7ff64c1422b0 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738935648 0 0) 0x7ff64c0accd0 con 0x7ff64c142690 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c071ee0 con 0x7ff64c142690 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2629333107 0 0) 0x7ff630004eb0 con 0x7ff64c1422b0 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff64c0accd0 con 0x7ff64c1422b0 2024-03-29T10:46:56.610 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff648001f80 con 0x7ff64c1422b0 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:56.611 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1009527685 0 0) 0x7ff64c0accd0 con 0x7ff64c1422b0 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 >> v1:172.21.15.94:6790/0 conn(0x7ff64c142690 legacy=0x7ff64c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.626+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 >> v1:172.21.15.173:6789/0 conn(0x7ff64c0b17d0 legacy=0x7ff64c0ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 1 monclient: found mon.a 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c0711e0 con 0x7ff64c1422b0 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff648004e10 con 0x7ff64c1422b0 2024-03-29T10:46:56.612 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff641ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:56.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.627+0000 7ff6437fe640 4 set_mon_vals no callback set 2024-03-29T10:46:56.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 5 monclient: authenticate success, global_id 4515 2024-03-29T10:46:56.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 10 monclient: _renew_subs 2024-03-29T10:46:56.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:56.614 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c0719a0 con 0x7ff64c1422b0 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 10 monclient: _renew_subs 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c1efa20 con 0x7ff64c1422b0 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.628+0000 7ff6539ba640 1 librados: init done 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.629+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff6480051a0 con 0x7ff64c1422b0 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.629+0000 7ff641ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.629+0000 7ff641ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.629+0000 7ff641ffb640 10 monclient: dump: 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:56.615 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.632+0000 7ff6539ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.632+0000 7ff6539ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.632+0000 7ff6539ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.632+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c075f10 con 0x7ff64c1422b0 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.633+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7ff648005300 con 0x7ff64c1422b0 2024-03-29T10:46:56.616 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.634+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7ff648084f30 con 0x7ff64c1422b0 2024-03-29T10:46:56.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.634+0000 7ff641ffb640 10 client.4515.objecter ms_dispatch 0x7ff64c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:56.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.634+0000 7ff641ffb640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:56.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.634+0000 7ff641ffb640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:56.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.634+0000 7ff641ffb640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T10:46:56.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.635+0000 7ff651756640 10 monclient: get_auth_request con 0x7ff63006dbd0 auth_method 0 2024-03-29T10:46:56.617 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.635+0000 7ff641ffb640 10 client.4515.objecter ms_handle_connect 0x7ff63006dbd0 2024-03-29T10:46:56.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.641+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff648050810 con 0x7ff64c1422b0 2024-03-29T10:46:56.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.641+0000 7ff641ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:56.623 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.641+0000 7ff641ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:56.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.976+0000 7ff6539ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:56.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.976+0000 7ff6539ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:56.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.976+0000 7ff6539ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:46:56.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.976+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff64c0b2270 con 0x7ff64c1422b0 2024-03-29T10:46:56.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.977+0000 7ff641ffb640 1 -- 172.21.15.94:0/2833699455 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7ff648052740 con 0x7ff64c1422b0 2024-03-29T10:46:56.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.977+0000 7ff641ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:56.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.977+0000 7ff641ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:56.960 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:46:56.960 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:56.962 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.980+0000 7ff6539ba640 10 librados: watch_flush enter 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.980+0000 7ff6539ba640 10 librados: watch_flush exit 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 10 client.4515.objecter successfully canceled tick 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 >> v1:172.21.15.94:6816/3829934808 conn(0x7ff63006dbd0 legacy=0x7ff630070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 10 monclient: shutdown 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 >> v1:172.21.15.94:6789/0 conn(0x7ff64c1422b0 legacy=0x7ff64c06f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 shutdown_connections 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.981+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 >> 172.21.15.94:0/2833699455 conn(0x7ff64c066a00 msgr2=0x7ff64c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:56.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.982+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 shutdown_connections 2024-03-29T10:46:56.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.982+0000 7ff6539ba640 1 -- 172.21.15.94:0/2833699455 wait complete. 2024-03-29T10:46:56.965 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:46:56.982+0000 7ff6539ba640 1 librados: shutdown 2024-03-29T10:46:56.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:03.788 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:03.942 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.960+0000 7ff117e14640 10 monclient: get_monmap_and_config 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.960+0000 7ff117e14640 10 monclient: build_initial_monmap 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.960+0000 7ff117e14640 10 monclient: monmap: 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:03.961483+0000 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:03.961483+0000 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:03.943 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:03.944 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.962+0000 7ff117e14640 1 Processor -- start 2024-03-29T10:47:03.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.962+0000 7ff117e14640 1 -- start start 2024-03-29T10:47:03.945 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.962+0000 7ff117e14640 10 monclient: init 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 10 monclient(hunting): picked mon.noname-c con 0x7ff1100b4710 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 10 monclient(hunting): picked mon.noname-a con 0x7ff1100b3970 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 10 monclient(hunting): picked mon.noname-b con 0x7ff11013f390 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100af020 con 0x7ff1100b3970 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100b0910 con 0x7ff1100b4710 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100b0bf0 con 0x7ff11013f390 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:03.948 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.963+0000 7ff117e14640 10 monclient(hunting): authenticate will time out at 1275.232064s 2024-03-29T10:47:03.949 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.967+0000 7ff115bb0640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff1100b4710 0x7ff1100b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:51101/0 (socket says 172.21.15.94:51101) 2024-03-29T10:47:03.950 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.967+0000 7ff115bb0640 1 -- 172.21.15.94:0/1996723865 learned_addr learned my addr 172.21.15.94:0/1996723865 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:03.950 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.968+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973199546 0 0) 0x7ff1100b0bf0 con 0x7ff11013f390 2024-03-29T10:47:03.950 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.968+0000 7ff114bae640 10 monclient(hunting): _negotiate 2024-03-29T10:47:03.950 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.968+0000 7ff114bae640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:03.950 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.968+0000 7ff114bae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:03.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.968+0000 7ff114bae640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:47:03.951 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.968+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc003520 con 0x7ff11013f390 2024-03-29T10:47:03.953 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109647487 0 0) 0x7ff1100b0910 con 0x7ff1100b4710 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient(hunting): _negotiate 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1100b0bf0 con 0x7ff1100b4710 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388281152 0 0) 0x7ff1100b0bf0 con 0x7ff1100b4710 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 >> v1:172.21.15.94:6790/0 conn(0x7ff11013f390 legacy=0x7ff11013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 >> v1:172.21.15.94:6789/0 conn(0x7ff1100b3970 legacy=0x7ff1100af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient: _finish_hunting 0 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 monclient: found mon.noname-c 2024-03-29T10:47:03.954 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1100aeb00 con 0x7ff1100b4710 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient: _finish_auth 0 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient: _check_auth_tickets 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff100001670 con 0x7ff1100b4710 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.969+0000 7ff114bae640 10 monclient: dump: 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:03.955 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.970+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1000017d0 con 0x7ff1100b4710 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.970+0000 7ff114bae640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.970+0000 7ff114bae640 1 -- 172.21.15.94:0/1996723865 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff100001ba0 con 0x7ff1100b4710 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.970+0000 7ff114bae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.970+0000 7ff114bae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.970+0000 7ff114bae640 10 monclient: dump: 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:03.956 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.971+0000 7ff117e14640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.971+0000 7ff117e14640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.971+0000 7ff117e14640 4 set_mon_vals no callback set 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.972+0000 7ff117e14640 10 monclient: shutdown 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.972+0000 7ff117e14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:03.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.972+0000 7ff117e14640 1 -- 172.21.15.94:0/1996723865 >> v1:172.21.15.173:6789/0 conn(0x7ff1100b4710 legacy=0x7ff1100b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:03.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.974+0000 7ff117e14640 1 -- 172.21.15.94:0/1996723865 shutdown_connections 2024-03-29T10:47:03.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.974+0000 7ff117e14640 1 -- 172.21.15.94:0/1996723865 >> 172.21.15.94:0/1996723865 conn(0x7ff110066a00 msgr2=0x7ff110066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:03.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.977+0000 7ff117e14640 1 -- 172.21.15.94:0/1996723865 shutdown_connections 2024-03-29T10:47:03.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.981+0000 7ff117e14640 1 -- 172.21.15.94:0/1996723865 wait complete. 2024-03-29T10:47:03.963 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.981+0000 7ff117e14640 10 monclient: build_initial_monmap 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.981+0000 7ff117e14640 10 monclient: monmap: 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:03.964 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.982+0000 7ff117e14640 1 librados: starting msgr at 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.982+0000 7ff117e14640 1 librados: starting objecter 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.982+0000 7ff117e14640 1 Processor -- start 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.982+0000 7ff117e14640 1 -- start start 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.982+0000 7ff117e14640 1 librados: setting wanted keys 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.982+0000 7ff117e14640 1 librados: calling monclient init 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.982+0000 7ff117e14640 10 monclient: init 2024-03-29T10:47:03.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.983+0000 7ff117e14640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:03.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.983+0000 7ff117e14640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:03.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 10 monclient(hunting): picked mon.noname-b con 0x7ff1100b3970 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:03.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 10 monclient(hunting): picked mon.noname-c con 0x7ff1100b4710 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:03.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 10 monclient(hunting): picked mon.noname-a con 0x7ff11013f390 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:03.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110071930 con 0x7ff11013f390 2024-03-29T10:47:03.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110071c10 con 0x7ff1100b3970 2024-03-29T10:47:03.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110071ef0 con 0x7ff1100b4710 2024-03-29T10:47:03.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:03.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.985+0000 7ff117e14640 10 monclient(hunting): authenticate will time out at 1275.254008s 2024-03-29T10:47:03.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.987+0000 7ff115bb0640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff1100b3970 0x7ff110070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:51267/0 (socket says 172.21.15.94:51267) 2024-03-29T10:47:03.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.987+0000 7ff115bb0640 1 -- 172.21.15.94:0/125842066 learned_addr learned my addr 172.21.15.94:0/125842066 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:03.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 client.?.objecter ms_handle_connect 0x7ff1100b3970 2024-03-29T10:47:03.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:03.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305333276 0 0) 0x7ff110071c10 con 0x7ff1100b3970 2024-03-29T10:47:03.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:03.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:03.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:03.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:47:03.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f4004eb0 con 0x7ff1100b3970 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3126284719 0 0) 0x7ff0f4004eb0 con 0x7ff1100b3970 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff110071c10 con 0x7ff1100b3970 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff100001c20 con 0x7ff1100b3970 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): dump: 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.973 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.989+0000 7ff1067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.990+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 35690078 0 0) 0x7ff110071c10 con 0x7ff1100b3970 2024-03-29T10:47:03.974 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.990+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 >> v1:172.21.15.94:6790/0 conn(0x7ff1100b4710 legacy=0x7ff110135040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 >> v1:172.21.15.94:6789/0 conn(0x7ff11013f390 legacy=0x7ff1101e6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 1 monclient: found mon.b 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11006fb70 con 0x7ff1100b3970 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:03.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 10 client.?.objecter ms_handle_connect 0x7ff11013f390 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff1067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 5 monclient: authenticate success, global_id 4477 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 10 monclient: _renew_subs 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff110070650 con 0x7ff1100b3970 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 10 monclient: _renew_subs 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11013b050 con 0x7ff1100b3970 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.991+0000 7ff117e14640 1 librados: init done 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.992+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff100004b10 con 0x7ff1100b3970 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.992+0000 7ff1067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:03.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.992+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff100004ee0 con 0x7ff1100b3970 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.992+0000 7ff1067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.992+0000 7ff1067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.992+0000 7ff1067fc640 10 monclient: dump: 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:03.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7ff100005040 con 0x7ff1100b3970 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff107fff640 4 set_mon_vals no callback set 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff117e14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7ff100084d10 con 0x7ff1100b3970 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff1067fc640 10 client.4477.objecter ms_dispatch 0x7ff110000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff1067fc640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff1067fc640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff1067fc640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff117e14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff117e14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.994+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f8005080 con 0x7ff1100b3970 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.995+0000 7ff1153af640 10 monclient: get_auth_request con 0x7ff0f406dbe0 auth_method 0 2024-03-29T10:47:03.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:03.995+0000 7ff1067fc640 10 client.4477.objecter ms_handle_connect 0x7ff0f406dbe0 2024-03-29T10:47:03.987 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.005+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1000505f0 con 0x7ff1100b3970 2024-03-29T10:47:03.987 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.005+0000 7ff1067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:03.987 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.005+0000 7ff1067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:04.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.330+0000 7ff117e14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:04.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.330+0000 7ff117e14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:04.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.330+0000 7ff117e14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:04.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.330+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0f8005330 con 0x7ff1100b3970 2024-03-29T10:47:04.316 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.334+0000 7ff1067fc640 1 -- 172.21.15.94:0/125842066 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7ff100052520 con 0x7ff1100b3970 2024-03-29T10:47:04.316 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.334+0000 7ff1067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:04.316 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.334+0000 7ff1067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:04.317 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:04.317 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:04.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.338+0000 7ff117e14640 10 librados: watch_flush enter 2024-03-29T10:47:04.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.338+0000 7ff117e14640 10 librados: watch_flush exit 2024-03-29T10:47:04.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.338+0000 7ff117e14640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T10:47:04.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.338+0000 7ff117e14640 10 client.4477.objecter successfully canceled tick 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.340+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 >> v1:172.21.15.94:6816/3829934808 conn(0x7ff0f406dbe0 legacy=0x7ff0f40700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.340+0000 7ff117e14640 10 monclient: shutdown 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.340+0000 7ff117e14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.340+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 >> v1:172.21.15.173:6789/0 conn(0x7ff1100b3970 legacy=0x7ff110070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.340+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 shutdown_connections 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.340+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 >> 172.21.15.94:0/125842066 conn(0x7ff110066a00 msgr2=0x7ff110069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.340+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 shutdown_connections 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.341+0000 7ff117e14640 1 -- 172.21.15.94:0/125842066 wait complete. 2024-03-29T10:47:04.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:04.341+0000 7ff117e14640 1 librados: shutdown 2024-03-29T10:47:04.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:11.147 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:11.296 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.314+0000 7f2641ef6640 10 monclient: get_monmap_and_config 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.314+0000 7f2641ef6640 10 monclient: build_initial_monmap 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.314+0000 7f2641ef6640 10 monclient: monmap: 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:11.315511+0000 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:11.315511+0000 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:11.297 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:11.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.315+0000 7f2641ef6640 1 Processor -- start 2024-03-29T10:47:11.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.315+0000 7f2641ef6640 1 -- start start 2024-03-29T10:47:11.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.315+0000 7f2641ef6640 10 monclient: init 2024-03-29T10:47:11.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:11.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:11.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f263c0b1f60 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:11.300 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f263c071eb0 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:11.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f263c143d10 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:11.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c06ef80 con 0x7f263c071eb0 2024-03-29T10:47:11.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c06f260 con 0x7f263c143d10 2024-03-29T10:47:11.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0b40c0 con 0x7f263c0b1f60 2024-03-29T10:47:11.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:11.301 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.316+0000 7f2641ef6640 10 monclient(hunting): authenticate will time out at 1282.584689s 2024-03-29T10:47:11.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.321+0000 7f263bfff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f263c143d10 0x7f263c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:39623/0 (socket says 172.21.15.94:39623) 2024-03-29T10:47:11.304 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.321+0000 7f263bfff640 1 -- 172.21.15.94:0/495617481 learned_addr learned my addr 172.21.15.94:0/495617481 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:11.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233756735 0 0) 0x7f263c06f260 con 0x7f263c143d10 2024-03-29T10:47:11.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:11.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:11.305 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2624003520 con 0x7f263c143d10 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744216922 0 0) 0x7f263c06ef80 con 0x7f263c071eb0 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.322+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263c06f260 con 0x7f263c071eb0 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648672475 0 0) 0x7f2624003520 con 0x7f263c143d10 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 >> v1:172.21.15.94:6790/0 conn(0x7f263c0b1f60 legacy=0x7f263c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 >> v1:172.21.15.94:6789/0 conn(0x7f263c071eb0 legacy=0x7f263c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.306 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c155d70 con 0x7f263c143d10 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f2641ef6640 5 monclient: authenticate success, global_id 4480 2024-03-29T10:47:11.307 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f2641ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2634001a10 con 0x7f263c143d10 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f263a7fc640 10 monclient: dump: 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:11.308 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.323+0000 7f2641ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2634001c60 con 0x7f263c143d10 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f263a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f263a7fc640 1 -- 172.21.15.94:0/495617481 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2634001ff0 con 0x7f263c143d10 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f263a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f263a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f263a7fc640 10 monclient: dump: 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:11.309 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f2641ef6640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:11.310 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.326+0000 7f2641ef6640 4 set_mon_vals no callback set 2024-03-29T10:47:11.311 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.327+0000 7f2641ef6640 10 monclient: shutdown 2024-03-29T10:47:11.311 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.327+0000 7f2641ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:11.311 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.327+0000 7f2641ef6640 1 -- 172.21.15.94:0/495617481 >> v1:172.21.15.173:6789/0 conn(0x7f263c143d10 legacy=0x7f263c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.311 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.327+0000 7f2641ef6640 1 -- 172.21.15.94:0/495617481 shutdown_connections 2024-03-29T10:47:11.311 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.327+0000 7f2641ef6640 1 -- 172.21.15.94:0/495617481 >> 172.21.15.94:0/495617481 conn(0x7f263c066a00 msgr2=0x7f263c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:11.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.331+0000 7f2641ef6640 1 -- 172.21.15.94:0/495617481 shutdown_connections 2024-03-29T10:47:11.313 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.331+0000 7f2641ef6640 1 -- 172.21.15.94:0/495617481 wait complete. 2024-03-29T10:47:11.317 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.332+0000 7f2641ef6640 10 monclient: build_initial_monmap 2024-03-29T10:47:11.317 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.332+0000 7f2641ef6640 10 monclient: monmap: 2024-03-29T10:47:11.317 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:11.317 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:11.317 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:11.317 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:11.317 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.332+0000 7f2641ef6640 1 librados: starting msgr at 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.332+0000 7f2641ef6640 1 librados: starting objecter 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.333+0000 7f2641ef6640 1 Processor -- start 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.333+0000 7f2641ef6640 1 -- start start 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.333+0000 7f2641ef6640 1 librados: setting wanted keys 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.333+0000 7f2641ef6640 1 librados: calling monclient init 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.333+0000 7f2641ef6640 10 monclient: init 2024-03-29T10:47:11.318 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f263c0b1f60 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f263c143d10 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f263c1df3d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c13bb90 con 0x7f263c0b1f60 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c14b530 con 0x7f263c143d10 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c149d00 con 0x7f263c1df3d0 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:11.319 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.334+0000 7f2641ef6640 10 monclient(hunting): authenticate will time out at 1282.602190s 2024-03-29T10:47:11.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.338+0000 7f263affd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f263c143d10 0x7f263c13ad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:39723/0 (socket says 172.21.15.94:39723) 2024-03-29T10:47:11.320 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.338+0000 7f263affd640 1 -- 172.21.15.94:0/1799324795 learned_addr learned my addr 172.21.15.94:0/1799324795 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 client.?.objecter ms_handle_connect 0x7f263c143d10 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 client.?.objecter ms_handle_connect 0x7f263c1df3d0 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970126555 0 0) 0x7f263c14b530 con 0x7f263c143d10 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.339+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2624003520 con 0x7f263c143d10 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2423808264 0 0) 0x7f2624003520 con 0x7f263c143d10 2024-03-29T10:47:11.323 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f263c14b530 con 0x7f263c143d10 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2630001a80 con 0x7f263c143d10 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 10 monclient(hunting): dump: 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:11.324 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:11.325 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1293181251 0 0) 0x7f263c14b530 con 0x7f263c143d10 2024-03-29T10:47:11.325 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 >> v1:172.21.15.94:6790/0 conn(0x7f263c1df3d0 legacy=0x7f263c13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.325 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.340+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 >> v1:172.21.15.94:6789/0 conn(0x7f263c0b1f60 legacy=0x7f263c1decc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 1 monclient: found mon.b 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c149990 con 0x7f263c143d10 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2623fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:47:11.328 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 10 monclient: _renew_subs 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c14a220 con 0x7f263c143d10 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 10 monclient: _renew_subs 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c154be0 con 0x7f263c143d10 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.342+0000 7f2641ef6640 1 librados: init done 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.343+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2630001be0 con 0x7f263c143d10 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.343+0000 7f2623fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.343+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f2630003ae0 con 0x7f263c143d10 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.343+0000 7f2623fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.343+0000 7f2623fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.343+0000 7f2623fff640 10 monclient: dump: 2024-03-29T10:47:11.329 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.343+0000 7f2639ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.344+0000 7f2641ef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.344+0000 7f2641ef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:11.330 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.344+0000 7f2641ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:11.331 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.344+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261c005080 con 0x7f263c143d10 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.346+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f2630003c40 con 0x7f263c143d10 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.346+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f2630084b90 con 0x7f263c143d10 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.346+0000 7f2623fff640 10 client.4483.objecter ms_dispatch 0x7f263c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.346+0000 7f2623fff640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.346+0000 7f2623fff640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.347+0000 7f2623fff640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.347+0000 7f263b7fe640 10 monclient: get_auth_request con 0x7f262406dde0 auth_method 0 2024-03-29T10:47:11.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.347+0000 7f2623fff640 10 client.4483.objecter ms_handle_connect 0x7f262406dde0 2024-03-29T10:47:11.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.355+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2630050470 con 0x7f263c143d10 2024-03-29T10:47:11.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.355+0000 7f2623fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:11.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.355+0000 7f2623fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:11.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.672+0000 7f2641ef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:11.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.672+0000 7f2641ef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:11.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.672+0000 7f2641ef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:11.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.672+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f261c005330 con 0x7f263c143d10 2024-03-29T10:47:11.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.675+0000 7f2623fff640 1 -- 172.21.15.94:0/1799324795 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f26300523a0 con 0x7f263c143d10 2024-03-29T10:47:11.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.675+0000 7f2623fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:11.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.675+0000 7f2623fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:11.659 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:11.659 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.678+0000 7f2641ef6640 10 librados: watch_flush enter 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.678+0000 7f2641ef6640 10 librados: watch_flush exit 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.678+0000 7f2641ef6640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.678+0000 7f2641ef6640 10 client.4483.objecter successfully canceled tick 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.679+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 >> v1:172.21.15.94:6816/3829934808 conn(0x7f262406dde0 legacy=0x7f26240702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.679+0000 7f2641ef6640 10 monclient: shutdown 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.679+0000 7f2641ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:11.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.679+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 >> v1:172.21.15.173:6789/0 conn(0x7f263c143d10 legacy=0x7f263c13ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:11.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.679+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 shutdown_connections 2024-03-29T10:47:11.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.679+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 >> 172.21.15.94:0/1799324795 conn(0x7f263c066a00 msgr2=0x7f263c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:11.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.680+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 shutdown_connections 2024-03-29T10:47:11.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.680+0000 7f2641ef6640 1 -- 172.21.15.94:0/1799324795 wait complete. 2024-03-29T10:47:11.664 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:11.681+0000 7f2641ef6640 1 librados: shutdown 2024-03-29T10:47:11.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:18.487 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:18.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.654+0000 7fe546d80640 10 monclient: get_monmap_and_config 2024-03-29T10:47:18.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.654+0000 7fe546d80640 10 monclient: build_initial_monmap 2024-03-29T10:47:18.637 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.654+0000 7fe546d80640 10 monclient: monmap: 2024-03-29T10:47:18.637 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:18.637 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:18.637 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:18.655796+0000 2024-03-29T10:47:18.637 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:18.655796+0000 2024-03-29T10:47:18.638 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:18.638 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:18.638 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:18.638 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:18.638 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:18.638 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:18.638 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.655+0000 7fe546d80640 1 Processor -- start 2024-03-29T10:47:18.641 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.659+0000 7fe546d80640 1 -- start start 2024-03-29T10:47:18.641 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.659+0000 7fe546d80640 10 monclient: init 2024-03-29T10:47:18.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.660+0000 7fe546d80640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:18.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.660+0000 7fe546d80640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:18.642 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.660+0000 7fe546d80640 10 monclient(hunting): picked mon.noname-c con 0x7fe5380389e0 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:18.645 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.660+0000 7fe546d80640 10 monclient(hunting): picked mon.noname-b con 0x7fe538038dc0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe546d80640 10 monclient(hunting): picked mon.noname-a con 0x7fe538030660 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe546d80640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5380c54c0 con 0x7fe538030660 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe546d80640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe538036430 con 0x7fe5380389e0 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe546d80640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe538039840 con 0x7fe538038dc0 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe546d80640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe546d80640 10 monclient(hunting): authenticate will time out at 1289.931597s 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe54557d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fe538038dc0 0x7fe5380cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35451/0 (socket says 172.21.15.94:35451) 2024-03-29T10:47:18.646 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.663+0000 7fe54557d640 1 -- 172.21.15.94:0/3599389069 learned_addr learned my addr 172.21.15.94:0/3599389069 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263392536 0 0) 0x7fe538039840 con 0x7fe538038dc0 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe52c003520 con 0x7fe538038dc0 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302539579 0 0) 0x7fe538036430 con 0x7fe5380389e0 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:47:18.650 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.667+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe538039840 con 0x7fe5380389e0 2024-03-29T10:47:18.653 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755779432 0 0) 0x7fe5380c54c0 con 0x7fe538030660 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe538036430 con 0x7fe538030660 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1168119619 0 0) 0x7fe538039840 con 0x7fe5380389e0 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 >> v1:172.21.15.94:6790/0 conn(0x7fe538038dc0 legacy=0x7fe5380cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 >> v1:172.21.15.94:6789/0 conn(0x7fe538030660 legacy=0x7fe5380d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient: _finish_hunting 0 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 1 monclient: found mon.noname-c 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:18.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538035690 con 0x7fe5380389e0 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient: _finish_auth 0 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 10 monclient: _check_auth_tickets 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.668+0000 7fe544d7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.669+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe53c0015f0 con 0x7fe5380389e0 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.669+0000 7fe544d7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.669+0000 7fe544d7c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.669+0000 7fe544d7c640 10 monclient: dump: 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:18.655 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:18.656 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:18.656 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:18.656 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:18.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.671+0000 7fe546d80640 5 monclient: authenticate success, global_id 4486 2024-03-29T10:47:18.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.671+0000 7fe546d80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:18.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.672+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe53c0017d0 con 0x7fe5380389e0 2024-03-29T10:47:18.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.672+0000 7fe544d7c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:18.657 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.672+0000 7fe544d7c640 1 -- 172.21.15.94:0/3599389069 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe53c0015f0 con 0x7fe5380389e0 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.672+0000 7fe544d7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.672+0000 7fe544d7c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.672+0000 7fe544d7c640 10 monclient: dump: 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:18.658 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.673+0000 7fe546d80640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.673+0000 7fe546d80640 4 set_mon_vals no callback set 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.674+0000 7fe546d80640 10 monclient: shutdown 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.674+0000 7fe546d80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.674+0000 7fe546d80640 1 -- 172.21.15.94:0/3599389069 >> v1:172.21.15.173:6789/0 conn(0x7fe5380389e0 legacy=0x7fe538033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.675+0000 7fe546d80640 1 -- 172.21.15.94:0/3599389069 shutdown_connections 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.675+0000 7fe546d80640 1 -- 172.21.15.94:0/3599389069 >> 172.21.15.94:0/3599389069 conn(0x7fe538005c00 msgr2=0x7fe538006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:18.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.677+0000 7fe546d80640 1 -- 172.21.15.94:0/3599389069 shutdown_connections 2024-03-29T10:47:18.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.677+0000 7fe546d80640 1 -- 172.21.15.94:0/3599389069 wait complete. 2024-03-29T10:47:18.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.678+0000 7fe546d80640 10 monclient: build_initial_monmap 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.678+0000 7fe546d80640 10 monclient: monmap: 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:18.661 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:18.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.679+0000 7fe546d80640 1 librados: starting msgr at 2024-03-29T10:47:18.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.679+0000 7fe546d80640 1 librados: starting objecter 2024-03-29T10:47:18.663 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.679+0000 7fe546d80640 1 Processor -- start 2024-03-29T10:47:18.663 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.679+0000 7fe546d80640 1 -- start start 2024-03-29T10:47:18.663 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.679+0000 7fe546d80640 1 librados: setting wanted keys 2024-03-29T10:47:18.663 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.679+0000 7fe546d80640 1 librados: calling monclient init 2024-03-29T10:47:18.663 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.679+0000 7fe546d80640 10 monclient: init 2024-03-29T10:47:18.663 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.680+0000 7fe546d80640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:18.663 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.680+0000 7fe546d80640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:18.666 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.680+0000 7fe546d80640 10 monclient(hunting): picked mon.noname-a con 0x7fe538030660 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:18.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.684+0000 7fe546d80640 10 monclient(hunting): picked mon.noname-c con 0x7fe5380389e0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:18.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.685+0000 7fe546d80640 10 monclient(hunting): picked mon.noname-b con 0x7fe5380c19b0 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:18.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.685+0000 7fe546d80640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53802f7c0 con 0x7fe538030660 2024-03-29T10:47:18.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.685+0000 7fe546d80640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53802fa30 con 0x7fe5380c19b0 2024-03-29T10:47:18.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.685+0000 7fe546d80640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53802e5c0 con 0x7fe5380389e0 2024-03-29T10:47:18.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.685+0000 7fe546d80640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:18.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.685+0000 7fe546d80640 10 monclient(hunting): authenticate will time out at 1289.953485s 2024-03-29T10:47:18.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.685+0000 7fe545d7e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fe538030660 0x7fe5380c12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42343/0 (socket says 172.21.15.94:42343) 2024-03-29T10:47:18.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.688+0000 7fe545d7e640 1 -- 172.21.15.94:0/1403152409 learned_addr learned my addr 172.21.15.94:0/1403152409 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:18.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.688+0000 7fe5367fc640 10 client.?.objecter ms_handle_connect 0x7fe538030660 2024-03-29T10:47:18.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.688+0000 7fe5367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:18.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.689+0000 7fe5367fc640 10 client.?.objecter ms_handle_connect 0x7fe5380389e0 2024-03-29T10:47:18.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.689+0000 7fe5367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:18.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.689+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984970477 0 0) 0x7fe53802e5c0 con 0x7fe5380389e0 2024-03-29T10:47:18.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.689+0000 7fe5367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.689+0000 7fe5367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.689+0000 7fe5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.689+0000 7fe5367fc640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:47:18.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.691+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe524004eb0 con 0x7fe5380389e0 2024-03-29T10:47:18.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.691+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928723645 0 0) 0x7fe53802f7c0 con 0x7fe538030660 2024-03-29T10:47:18.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.691+0000 7fe5367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.691+0000 7fe5367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.691+0000 7fe5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.691+0000 7fe5367fc640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:47:18.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.691+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53802e5c0 con 0x7fe538030660 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 client.?.objecter ms_handle_connect 0x7fe5380c19b0 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1506570039 0 0) 0x7fe524004eb0 con 0x7fe5380389e0 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe53802f7c0 con 0x7fe5380389e0 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907642522 0 0) 0x7fe53802fa30 con 0x7fe5380c19b0 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe524004eb0 con 0x7fe5380c19b0 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe5300017b0 con 0x7fe5380389e0 2024-03-29T10:47:18.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): dump: 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 92627317 0 0) 0x7fe524004eb0 con 0x7fe5380c19b0 2024-03-29T10:47:18.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe53802fa30 con 0x7fe5380c19b0 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe54004fdd0 con 0x7fe5380c19b0 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.692+0000 7fe5367fc640 10 monclient(hunting): dump: 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:18.677 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1572350491 0 0) 0x7fe53802f7c0 con 0x7fe5380389e0 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 >> v1:172.21.15.173:6789/0 conn(0x7fe5380c19b0 legacy=0x7fe538158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 >> v1:172.21.15.94:6789/0 conn(0x7fe538030660 legacy=0x7fe5380c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 1 monclient: found mon.c 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53802dc10 con 0x7fe5380389e0 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe5367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:18.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 10 monclient: _renew_subs 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53802fe50 con 0x7fe5380389e0 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 10 monclient: _renew_subs 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53802d640 con 0x7fe5380389e0 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.693+0000 7fe546d80640 1 librados: init done 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe530001c60 con 0x7fe5380389e0 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe5367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe530004980 con 0x7fe5380389e0 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe5367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe5367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe5367fc640 10 monclient: dump: 2024-03-29T10:47:18.681 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fe530004ba0 con 0x7fe5380389e0 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe546d80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe546d80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe546d80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:18.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.696+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe538007b90 con 0x7fe5380389e0 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.697+0000 7fe537fff640 4 set_mon_vals no callback set 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.697+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fe530084a80 con 0x7fe5380389e0 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.697+0000 7fe5367fc640 10 client.4547.objecter ms_dispatch 0x7fe540000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.697+0000 7fe5367fc640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.697+0000 7fe5367fc640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.697+0000 7fe5367fc640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.697+0000 7fe545d7e640 10 monclient: get_auth_request con 0x7fe52406df20 auth_method 0 2024-03-29T10:47:18.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.698+0000 7fe5367fc640 10 client.4547.objecter ms_handle_connect 0x7fe52406df20 2024-03-29T10:47:18.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.707+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5300503e0 con 0x7fe5380389e0 2024-03-29T10:47:18.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.707+0000 7fe5367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:18.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:18.707+0000 7fe5367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:19.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.029+0000 7fe546d80640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:19.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.029+0000 7fe546d80640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:19.011 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.029+0000 7fe546d80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:19.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.029+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe53802e750 con 0x7fe5380389e0 2024-03-29T10:47:19.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.030+0000 7fe5367fc640 1 -- 172.21.15.94:0/1403152409 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fe530052310 con 0x7fe5380389e0 2024-03-29T10:47:19.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.030+0000 7fe5367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:19.012 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.030+0000 7fe5367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:19.012 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:19.013 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:19.015 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.033+0000 7fe546d80640 10 librados: watch_flush enter 2024-03-29T10:47:19.015 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 10 librados: watch_flush exit 2024-03-29T10:47:19.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T10:47:19.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 10 client.4547.objecter successfully canceled tick 2024-03-29T10:47:19.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 >> v1:172.21.15.94:6816/3829934808 conn(0x7fe52406df20 legacy=0x7fe5240703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:19.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 10 monclient: shutdown 2024-03-29T10:47:19.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:19.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 >> v1:172.21.15.94:6790/0 conn(0x7fe5380389e0 legacy=0x7fe538158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:19.016 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 shutdown_connections 2024-03-29T10:47:19.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 >> 172.21.15.94:0/1403152409 conn(0x7fe538005c00 msgr2=0x7fe5380c49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:19.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 shutdown_connections 2024-03-29T10:47:19.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 1 -- 172.21.15.94:0/1403152409 wait complete. 2024-03-29T10:47:19.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:19.034+0000 7fe546d80640 1 librados: shutdown 2024-03-29T10:47:19.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:25.841 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:26.002 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.019+0000 7fe8a175e640 10 monclient: get_monmap_and_config 2024-03-29T10:47:26.002 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.019+0000 7fe8a175e640 10 monclient: build_initial_monmap 2024-03-29T10:47:26.002 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.019+0000 7fe8a175e640 10 monclient: monmap: 2024-03-29T10:47:26.002 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:26.002 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:26.019934+0000 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:26.019934+0000 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:26.003 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.019+0000 7fe8a175e640 1 Processor -- start 2024-03-29T10:47:26.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.020+0000 7fe8a175e640 1 -- start start 2024-03-29T10:47:26.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.020+0000 7fe8a175e640 10 monclient: init 2024-03-29T10:47:26.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:26.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:26.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 10 monclient(hunting): picked mon.noname-c con 0x7fe89c0b0860 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:26.017 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 10 monclient(hunting): picked mon.noname-a con 0x7fe89c13ef30 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 10 monclient(hunting): picked mon.noname-b con 0x7fe89c0af8e0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe89affd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe89c0b0860 0x7fe89c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:40396/0 (socket says 172.21.15.94:40396) 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe89affd640 1 -- 172.21.15.94:0/2562222158 learned_addr learned my addr 172.21.15.94:0/2562222158 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0b0c40 con 0x7fe89c13ef30 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0aed10 con 0x7fe89c0b0860 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.021+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0afd50 con 0x7fe89c0af8e0 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe8a175e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe8a175e640 10 monclient(hunting): authenticate will time out at 1297.290229s 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116765667 0 0) 0x7fe89c0aed10 con 0x7fe89c0b0860 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:47:26.018 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe884003520 con 0x7fe89c0b0860 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162049922 0 0) 0x7fe89c0b0c40 con 0x7fe89c13ef30 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe89c0aed10 con 0x7fe89c13ef30 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792442239 0 0) 0x7fe89c0afd50 con 0x7fe89c0af8e0 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe89c0b0c40 con 0x7fe89c0af8e0 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3285824043 0 0) 0x7fe89c0aed10 con 0x7fe89c13ef30 2024-03-29T10:47:26.019 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.022+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 >> v1:172.21.15.94:6790/0 conn(0x7fe89c0af8e0 legacy=0x7fe89c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 >> v1:172.21.15.173:6789/0 conn(0x7fe89c0b0860 legacy=0x7fe89c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 1 monclient: found mon.noname-a 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0ab1b0 con 0x7fe89c13ef30 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe890001670 con 0x7fe89c13ef30 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: dump: 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:26.020 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe8a175e640 5 monclient: authenticate success, global_id 4527 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe8a175e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8900017d0 con 0x7fe89c13ef30 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 1 -- 172.21.15.94:0/2562222158 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe890001ba0 con 0x7fe89c13ef30 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.021 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe899ffb640 10 monclient: dump: 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe8a175e640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.023+0000 7fe8a175e640 4 set_mon_vals no callback set 2024-03-29T10:47:26.022 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.024+0000 7fe8a175e640 10 monclient: shutdown 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.024+0000 7fe8a175e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.024+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 >> v1:172.21.15.94:6789/0 conn(0x7fe89c13ef30 legacy=0x7fe89c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.024+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 shutdown_connections 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.024+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 >> 172.21.15.94:0/2562222158 conn(0x7fe89c06b660 msgr2=0x7fe89c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.024+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 shutdown_connections 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.025+0000 7fe8a175e640 1 -- 172.21.15.94:0/2562222158 wait complete. 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.025+0000 7fe8a175e640 10 monclient: build_initial_monmap 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.025+0000 7fe8a175e640 10 monclient: monmap: 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:26.023 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.026+0000 7fe8a175e640 1 librados: starting msgr at 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.026+0000 7fe8a175e640 1 librados: starting objecter 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.027+0000 7fe8a175e640 1 Processor -- start 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.027+0000 7fe8a175e640 1 -- start start 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.027+0000 7fe8a175e640 1 librados: setting wanted keys 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.027+0000 7fe8a175e640 1 librados: calling monclient init 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.027+0000 7fe8a175e640 10 monclient: init 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 10 monclient(hunting): picked mon.noname-b con 0x7fe89c0af8e0 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 10 monclient(hunting): picked mon.noname-c con 0x7fe89c0b0860 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:26.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 10 monclient(hunting): picked mon.noname-a con 0x7fe89c13ef30 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe89affd640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fe89c0af8e0 0x7fe89c0b2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:41440/0 (socket says 172.21.15.94:41440) 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0b3610 con 0x7fe89c13ef30 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe89affd640 1 -- 172.21.15.94:0/3644733729 learned_addr learned my addr 172.21.15.94:0/3644733729 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c1c4910 con 0x7fe89c0af8e0 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8837fe640 10 client.?.objecter ms_handle_connect 0x7fe89c0af8e0 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0b31d0 con 0x7fe89c0b0860 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.028+0000 7fe8a175e640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8a175e640 10 monclient(hunting): authenticate will time out at 1297.297105s 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 client.?.objecter ms_handle_connect 0x7fe89c0b0860 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 client.?.objecter ms_handle_connect 0x7fe89c13ef30 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:26.025 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366670050 0 0) 0x7fe89c0b31d0 con 0x7fe89c0b0860 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe888004eb0 con 0x7fe89c0b0860 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515776923 0 0) 0x7fe89c1c4910 con 0x7fe89c0af8e0 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.029+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe89c0b31d0 con 0x7fe89c0af8e0 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449058284 0 0) 0x7fe89c0b3610 con 0x7fe89c13ef30 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:26.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe89c1c4910 con 0x7fe89c13ef30 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3629191584 0 0) 0x7fe89c0b31d0 con 0x7fe89c0af8e0 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe89c0b3610 con 0x7fe89c0af8e0 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe88c001920 con 0x7fe89c0af8e0 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): dump: 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.027 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3797628044 0 0) 0x7fe888004eb0 con 0x7fe89c0b0860 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe89c0b31d0 con 0x7fe89c0b0860 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe890001b60 con 0x7fe89c0b0860 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.030+0000 7fe8837fe640 10 monclient(hunting): dump: 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.028 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1225981248 0 0) 0x7fe89c0b3610 con 0x7fe89c0af8e0 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 >> v1:172.21.15.94:6790/0 conn(0x7fe89c0b0860 legacy=0x7fe89c132780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 >> v1:172.21.15.94:6789/0 conn(0x7fe89c13ef30 legacy=0x7fe89c1d2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 1 monclient: found mon.b 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c132e90 con 0x7fe89c0af8e0 2024-03-29T10:47:26.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 5 monclient: authenticate success, global_id 4495 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 10 monclient: _renew_subs 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c1330e0 con 0x7fe89c0af8e0 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 10 monclient: _renew_subs 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c137320 con 0x7fe89c0af8e0 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.031+0000 7fe8a175e640 1 librados: init done 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.032+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe88c0047e0 con 0x7fe89c0af8e0 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8a175e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8a175e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:26.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8a175e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c075f10 con 0x7fe89c0af8e0 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fe88c004af0 con 0x7fe89c0af8e0 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe8837fe640 10 monclient: dump: 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:26.031 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:26.032 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:26.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.033+0000 7fe898ff9640 4 set_mon_vals no callback set 2024-03-29T10:47:26.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.035+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fe88c01aac0 con 0x7fe89c0af8e0 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.036+0000 7fe89a7fc640 10 monclient: get_auth_request con 0x7fe88806df70 auth_method 0 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.036+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fe88c084a50 con 0x7fe89c0af8e0 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.036+0000 7fe8837fe640 10 client.4495.objecter ms_dispatch 0x7fe89c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.037+0000 7fe8837fe640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.037+0000 7fe8837fe640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.037+0000 7fe8837fe640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.037+0000 7fe8837fe640 10 client.4495.objecter ms_handle_connect 0x7fe88806df70 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.043+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe88c050330 con 0x7fe89c0af8e0 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.043+0000 7fe8837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:26.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.043+0000 7fe8837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:26.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.428+0000 7fe8a175e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:26.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.428+0000 7fe8a175e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:26.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.428+0000 7fe8a175e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:26.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.428+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe89c072330 con 0x7fe89c0af8e0 2024-03-29T10:47:26.415 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:26.415 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:26.416 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.430+0000 7fe8837fe640 1 -- 172.21.15.94:0/3644733729 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fe88c052260 con 0x7fe89c0af8e0 2024-03-29T10:47:26.416 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.430+0000 7fe8837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:26.416 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.430+0000 7fe8837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:26.420 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.436+0000 7fe8a175e640 10 librados: watch_flush enter 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.439+0000 7fe8a175e640 10 librados: watch_flush exit 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.439+0000 7fe8a175e640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.440+0000 7fe8a175e640 10 client.4495.objecter successfully canceled tick 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.440+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 >> v1:172.21.15.94:6816/3829934808 conn(0x7fe88806df70 legacy=0x7fe888070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.440+0000 7fe8a175e640 10 monclient: shutdown 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.440+0000 7fe8a175e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.440+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 >> v1:172.21.15.173:6789/0 conn(0x7fe89c0af8e0 legacy=0x7fe89c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.440+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 shutdown_connections 2024-03-29T10:47:26.423 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.440+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 >> 172.21.15.94:0/3644733729 conn(0x7fe89c06b660 msgr2=0x7fe89c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:26.424 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.441+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 shutdown_connections 2024-03-29T10:47:26.424 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.441+0000 7fe8a175e640 1 -- 172.21.15.94:0/3644733729 wait complete. 2024-03-29T10:47:26.424 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:26.441+0000 7fe8a175e640 1 librados: shutdown 2024-03-29T10:47:26.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:33.250 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:33.424 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.441+0000 7fc0328ae640 10 monclient: get_monmap_and_config 2024-03-29T10:47:33.424 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.441+0000 7fc0328ae640 10 monclient: build_initial_monmap 2024-03-29T10:47:33.424 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.441+0000 7fc0328ae640 10 monclient: monmap: 2024-03-29T10:47:33.424 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:33.443109+0000 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:33.443109+0000 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:33.425 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:33.426 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.442+0000 7fc0328ae640 1 Processor -- start 2024-03-29T10:47:33.426 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.442+0000 7fc0328ae640 1 -- start start 2024-03-29T10:47:33.426 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.442+0000 7fc0328ae640 10 monclient: init 2024-03-29T10:47:33.426 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc02c0b0770 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc02c14a700 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc02c0ac5a0 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0aba70 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0aecb0 con 0x7fc02c0b0770 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0af150 con 0x7fc02c14a700 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc0328ae640 10 monclient(hunting): authenticate will time out at 1304.712832s 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.443+0000 7fc02b7fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fc02c14a700 0x7fc02c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35964/0 (socket says 172.21.15.94:35964) 2024-03-29T10:47:33.427 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02b7fe640 1 -- 172.21.15.94:0/4145862723 learned_addr learned my addr 172.21.15.94:0/4145862723 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:33.428 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027905313 0 0) 0x7fc02c0af150 con 0x7fc02c14a700 2024-03-29T10:47:33.428 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.428 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02affd640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014003520 con 0x7fc02c14a700 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.444+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50161950 0 0) 0x7fc02c0aecb0 con 0x7fc02c0b0770 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc02c0af150 con 0x7fc02c0b0770 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936040347 0 0) 0x7fc02c0aba70 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.429 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.430 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:47:33.430 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc02c0aecb0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.430 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058993747 0 0) 0x7fc014003520 con 0x7fc02c14a700 2024-03-29T10:47:33.430 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.445+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 >> v1:172.21.15.173:6789/0 conn(0x7fc02c0b0770 legacy=0x7fc02c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.431 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 >> v1:172.21.15.94:6789/0 conn(0x7fc02c0ac5a0 legacy=0x7fc02c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 1 monclient: found mon.noname-b 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0ac9d0 con 0x7fc02c14a700 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 10 monclient: _finish_auth 0 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc0328ae640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc0328ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc020001690 con 0x7fc02c14a700 2024-03-29T10:47:33.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc02affd640 10 monclient: dump: 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:33.433 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.446+0000 7fc0328ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0200017f0 con 0x7fc02c14a700 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc02affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc02affd640 1 -- 172.21.15.94:0/4145862723 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc020001c40 con 0x7fc02c14a700 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc0328ae640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc0328ae640 4 set_mon_vals no callback set 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc02affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc02affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.447+0000 7fc02affd640 10 monclient: dump: 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:33.434 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.448+0000 7fc0328ae640 10 monclient: shutdown 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.448+0000 7fc0328ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:33.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.448+0000 7fc0328ae640 1 -- 172.21.15.94:0/4145862723 >> v1:172.21.15.94:6790/0 conn(0x7fc02c14a700 legacy=0x7fc02c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.448+0000 7fc0328ae640 1 -- 172.21.15.94:0/4145862723 shutdown_connections 2024-03-29T10:47:33.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.448+0000 7fc0328ae640 1 -- 172.21.15.94:0/4145862723 >> 172.21.15.94:0/4145862723 conn(0x7fc02c06b660 msgr2=0x7fc02c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:33.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.448+0000 7fc0328ae640 1 -- 172.21.15.94:0/4145862723 shutdown_connections 2024-03-29T10:47:33.436 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.448+0000 7fc0328ae640 1 -- 172.21.15.94:0/4145862723 wait complete. 2024-03-29T10:47:33.441 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.449+0000 7fc0328ae640 10 monclient: build_initial_monmap 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.449+0000 7fc0328ae640 10 monclient: monmap: 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:33.442 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.450+0000 7fc0328ae640 1 librados: starting msgr at 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.450+0000 7fc0328ae640 1 librados: starting objecter 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.450+0000 7fc0328ae640 1 Processor -- start 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.450+0000 7fc0328ae640 1 -- start start 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.450+0000 7fc0328ae640 1 librados: setting wanted keys 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.450+0000 7fc0328ae640 1 librados: calling monclient init 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.450+0000 7fc0328ae640 10 monclient: init 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.451+0000 7fc0328ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.451+0000 7fc0328ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.451+0000 7fc0328ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc02c0ac5a0 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:33.443 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.451+0000 7fc0328ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc02c0b0770 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.451+0000 7fc0328ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc02c126ea0 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.451+0000 7fc0328ae640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c1239d0 con 0x7fc02c126ea0 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc0328ae640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c127fc0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc0328ae640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c128230 con 0x7fc02c0b0770 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc0328ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc0328ae640 10 monclient(hunting): authenticate will time out at 1304.721147s 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc02bfff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc02c0ac5a0 0x7fc02c144e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:58192/0 (socket says 172.21.15.94:58192) 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc02bfff640 1 -- 172.21.15.94:0/546991930 learned_addr learned my addr 172.21.15.94:0/546991930 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 client.?.objecter ms_handle_connect 0x7fc02c0ac5a0 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 client.?.objecter ms_handle_connect 0x7fc02c126ea0 2024-03-29T10:47:33.444 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563925821 0 0) 0x7fc02c127fc0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 10 monclient(hunting): my global_id is 4501 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.452+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc018004eb0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3277905806 0 0) 0x7fc018004eb0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc02c127fc0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc01c001950 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:33.445 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:33.446 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278439259 0 0) 0x7fc02c1239d0 con 0x7fc02c126ea0 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc018004eb0 con 0x7fc02c126ea0 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 client.?.objecter ms_handle_connect 0x7fc02c0b0770 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 875735306 0 0) 0x7fc02c127fc0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 >> v1:172.21.15.94:6790/0 conn(0x7fc02c0b0770 legacy=0x7fc02c1dbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 >> v1:172.21.15.94:6789/0 conn(0x7fc02c126ea0 legacy=0x7fc02c1e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 monclient: found mon.b 2024-03-29T10:47:33.447 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c125eb0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc028ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc0328ae640 5 monclient: authenticate success, global_id 4501 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.453+0000 7fc0328ae640 10 monclient: _renew_subs 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc0328ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c1263d0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc0328ae640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc0328ae640 10 monclient: _renew_subs 2024-03-29T10:47:33.448 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc0328ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c145530 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc0328ae640 1 librados: init done 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc01c0035f0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc028ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fc01c0039c0 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc028ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc028ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc028ff9640 10 monclient: dump: 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.449 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.454+0000 7fc02a7fc640 4 set_mon_vals no callback set 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.456+0000 7fc0328ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.456+0000 7fc0328ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.456+0000 7fc0328ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.456+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c005080 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.450 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.457+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fc01c003b20 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.457+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fc01c084a50 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.457+0000 7fc028ff9640 10 client.4501.objecter ms_dispatch 0x7fc02c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.457+0000 7fc028ff9640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.457+0000 7fc028ff9640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.457+0000 7fc028ff9640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.458+0000 7fc02b7fe640 10 monclient: get_auth_request con 0x7fc01806deb0 auth_method 0 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.458+0000 7fc028ff9640 10 client.4501.objecter ms_handle_connect 0x7fc01806deb0 2024-03-29T10:47:33.451 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.463+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc01c050330 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.463+0000 7fc028ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:33.452 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.463+0000 7fc028ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:33.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.895+0000 7fc0328ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:33.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.895+0000 7fc0328ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:33.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.895+0000 7fc0328ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:33.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.895+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc00c005330 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.897+0000 7fc028ff9640 1 -- 172.21.15.94:0/546991930 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fc01c052260 con 0x7fc02c0ac5a0 2024-03-29T10:47:33.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.897+0000 7fc028ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:33.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.897+0000 7fc028ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:33.881 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:33.881 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 10 librados: watch_flush enter 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 10 librados: watch_flush exit 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 10 client.4501.objecter successfully canceled tick 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 >> v1:172.21.15.94:6816/3829934808 conn(0x7fc01806deb0 legacy=0x7fc018070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 10 monclient: shutdown 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 >> v1:172.21.15.173:6789/0 conn(0x7fc02c0ac5a0 legacy=0x7fc02c144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:33.885 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 shutdown_connections 2024-03-29T10:47:33.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.901+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 >> 172.21.15.94:0/546991930 conn(0x7fc02c06b660 msgr2=0x7fc02c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:33.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.902+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 shutdown_connections 2024-03-29T10:47:33.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.902+0000 7fc0328ae640 1 -- 172.21.15.94:0/546991930 wait complete. 2024-03-29T10:47:33.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:33.902+0000 7fc0328ae640 1 librados: shutdown 2024-03-29T10:47:33.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:40.708 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:40.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.873+0000 7f6ec92a8640 10 monclient: get_monmap_and_config 2024-03-29T10:47:40.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.874+0000 7f6ec92a8640 10 monclient: build_initial_monmap 2024-03-29T10:47:40.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.874+0000 7f6ec92a8640 10 monclient: monmap: 2024-03-29T10:47:40.858 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:40.858 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:40.875898+0000 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:40.875898+0000 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:40.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.875+0000 7f6ec92a8640 1 Processor -- start 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.875+0000 7f6ec92a8640 1 -- start start 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.875+0000 7f6ec92a8640 10 monclient: init 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ebc034850 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ebc02a640 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ebc0327d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc02c820 con 0x7f6ebc034850 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc02a120 con 0x7f6ebc02a640 2024-03-29T10:47:40.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc0c75b0 con 0x7f6ebc0327d0 2024-03-29T10:47:40.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:40.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.876+0000 7f6ec92a8640 10 monclient(hunting): authenticate will time out at 1312.145282s 2024-03-29T10:47:40.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.879+0000 7f6ec2ffd640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6ebc034850 0x7f6ebc0c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:42334/0 (socket says 172.21.15.94:42334) 2024-03-29T10:47:40.863 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.879+0000 7f6ec2ffd640 1 -- 172.21.15.94:0/349796166 learned_addr learned my addr 172.21.15.94:0/349796166 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278058124 0 0) 0x7f6ebc02c820 con 0x7f6ebc034850 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eac003520 con 0x7f6ebc034850 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434955372 0 0) 0x7f6eac003520 con 0x7f6ebc034850 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 >> v1:172.21.15.94:6790/0 conn(0x7f6ebc0327d0 legacy=0x7f6ebc0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:40.864 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.880+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 >> v1:172.21.15.173:6789/0 conn(0x7f6ebc02a640 legacy=0x7f6ebc0320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 1 monclient: found mon.noname-a 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc033110 con 0x7f6ebc034850 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:40.867 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6eb8001690 con 0x7f6ebc034850 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: dump: 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:40.868 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6eb8001920 con 0x7f6ebc034850 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 1 -- 172.21.15.94:0/349796166 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6eb8001cb0 con 0x7f6ebc034850 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.884+0000 7f6ec1ffb640 10 monclient: dump: 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:40.869 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.885+0000 7f6ec92a8640 5 monclient: authenticate success, global_id 4539 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.885+0000 7f6ec92a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.885+0000 7f6ec92a8640 4 set_mon_vals no callback set 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.886+0000 7f6ec92a8640 10 monclient: shutdown 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.886+0000 7f6ec92a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:40.870 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.886+0000 7f6ec92a8640 1 -- 172.21.15.94:0/349796166 >> v1:172.21.15.94:6789/0 conn(0x7f6ebc034850 legacy=0x7f6ebc0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:40.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.888+0000 7f6ec92a8640 1 -- 172.21.15.94:0/349796166 shutdown_connections 2024-03-29T10:47:40.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.888+0000 7f6ec92a8640 1 -- 172.21.15.94:0/349796166 >> 172.21.15.94:0/349796166 conn(0x7f6ebc0058f0 msgr2=0x7f6ebc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:40.871 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.888+0000 7f6ec92a8640 1 -- 172.21.15.94:0/349796166 shutdown_connections 2024-03-29T10:47:40.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.888+0000 7f6ec92a8640 1 -- 172.21.15.94:0/349796166 wait complete. 2024-03-29T10:47:40.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.889+0000 7f6ec92a8640 10 monclient: build_initial_monmap 2024-03-29T10:47:40.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.889+0000 7f6ec92a8640 10 monclient: monmap: 2024-03-29T10:47:40.873 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.889+0000 7f6ec92a8640 1 librados: starting msgr at 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.889+0000 7f6ec92a8640 1 librados: starting objecter 2024-03-29T10:47:40.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.890+0000 7f6ec92a8640 1 Processor -- start 2024-03-29T10:47:40.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.890+0000 7f6ec92a8640 1 -- start start 2024-03-29T10:47:40.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.890+0000 7f6ec92a8640 1 librados: setting wanted keys 2024-03-29T10:47:40.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.890+0000 7f6ec92a8640 1 librados: calling monclient init 2024-03-29T10:47:40.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.890+0000 7f6ec92a8640 10 monclient: init 2024-03-29T10:47:40.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:40.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:40.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ebc02a640 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ebc0327d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ebc034850 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc0c4250 con 0x7f6ebc034850 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc02e890 con 0x7f6ebc02a640 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc1650b0 con 0x7f6ebc0327d0 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.891+0000 7f6ec92a8640 10 monclient(hunting): authenticate will time out at 1312.160863s 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.893+0000 7f6ec37fe640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6ebc034850 0x7f6ebc1649a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:43576/0 (socket says 172.21.15.94:43576) 2024-03-29T10:47:40.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.893+0000 7f6ec37fe640 1 -- 172.21.15.94:0/385227214 learned_addr learned my addr 172.21.15.94:0/385227214 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:40.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 client.?.objecter ms_handle_connect 0x7f6ebc02a640 2024-03-29T10:47:40.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:40.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256460431 0 0) 0x7f6ebc02e890 con 0x7f6ebc02a640 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb0004eb0 con 0x7f6ebc02a640 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 client.?.objecter ms_handle_connect 0x7f6ebc034850 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.896+0000 7f6eab7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503371436 0 0) 0x7f6ebc0c4250 con 0x7f6ebc034850 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:40.881 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ebc02e890 con 0x7f6ebc034850 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2812188528 0 0) 0x7f6eb0004eb0 con 0x7f6ebc02a640 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ebc0c4250 con 0x7f6ebc02a640 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6eb8001550 con 0x7f6ebc02a640 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): dump: 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:40.882 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3671694850 0 0) 0x7f6ebc0c4250 con 0x7f6ebc02a640 2024-03-29T10:47:40.883 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.897+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 >> v1:172.21.15.94:6790/0 conn(0x7f6ebc0327d0 legacy=0x7f6ebc164290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:40.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 >> v1:172.21.15.94:6789/0 conn(0x7f6ebc034850 legacy=0x7f6ebc1649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:40.886 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 1 monclient: found mon.b 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc02d4c0 con 0x7f6ebc02a640 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6eab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6ec92a8640 5 monclient: authenticate success, global_id 4507 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6ec92a8640 10 monclient: _renew_subs 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6ec92a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.898+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc02d7a0 con 0x7f6ebc02a640 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6ec92a8640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:40.887 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6ec92a8640 10 monclient: _renew_subs 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6ec92a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc0c3790 con 0x7f6ebc02a640 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6ec92a8640 1 librados: init done 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6eb80016b0 con 0x7f6ebc02a640 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6eab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.899+0000 7f6ec17fa640 4 set_mon_vals no callback set 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f6eb8004c30 con 0x7f6ebc02a640 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6eab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6eab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6eab7fe640 10 monclient: dump: 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:40.888 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6ec92a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6ec92a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6ec92a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.900+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea4005080 con 0x7f6ebc02a640 2024-03-29T10:47:40.889 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.903+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f6eb8004d90 con 0x7f6ebc02a640 2024-03-29T10:47:40.890 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.905+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f6eb8084ad0 con 0x7f6ebc02a640 2024-03-29T10:47:40.890 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.905+0000 7f6eab7fe640 10 client.4507.objecter ms_dispatch 0x7f6ec4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:40.890 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.905+0000 7f6eab7fe640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:40.890 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.905+0000 7f6eab7fe640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:40.890 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.905+0000 7f6eab7fe640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T10:47:40.890 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.907+0000 7f6ec27fc640 10 monclient: get_auth_request con 0x7f6eb006d970 auth_method 0 2024-03-29T10:47:40.891 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.907+0000 7f6eab7fe640 10 client.4507.objecter ms_handle_connect 0x7f6eb006d970 2024-03-29T10:47:40.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.911+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6eb8086e60 con 0x7f6ebc02a640 2024-03-29T10:47:40.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.911+0000 7f6eab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:40.894 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:40.911+0000 7f6eab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:41.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.238+0000 7f6ec92a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:41.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.238+0000 7f6ec92a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:41.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.238+0000 7f6ec92a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:47:41.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.238+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ea4005330 con 0x7f6ebc02a640 2024-03-29T10:47:41.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.245+0000 7f6eab7fe640 1 -- 172.21.15.94:0/385227214 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f6eb80501e0 con 0x7f6ebc02a640 2024-03-29T10:47:41.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.245+0000 7f6eab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:41.228 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.245+0000 7f6eab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:41.229 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:41.229 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:41.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.249+0000 7f6ec92a8640 10 librados: watch_flush enter 2024-03-29T10:47:41.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 10 librados: watch_flush exit 2024-03-29T10:47:41.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T10:47:41.233 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 10 client.4507.objecter successfully canceled tick 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 >> v1:172.21.15.94:6816/3829934808 conn(0x7f6eb006d970 legacy=0x7f6eb006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 10 monclient: shutdown 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 >> v1:172.21.15.173:6789/0 conn(0x7f6ebc02a640 legacy=0x7f6ebc02e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 shutdown_connections 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.250+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 >> 172.21.15.94:0/385227214 conn(0x7f6ebc0058f0 msgr2=0x7f6ebc0320c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.251+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 shutdown_connections 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.251+0000 7f6ec92a8640 1 -- 172.21.15.94:0/385227214 wait complete. 2024-03-29T10:47:41.234 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:41.251+0000 7f6ec92a8640 1 librados: shutdown 2024-03-29T10:47:41.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:48.057 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:48.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.212+0000 7fcbf71bc640 10 monclient: get_monmap_and_config 2024-03-29T10:47:48.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.212+0000 7fcbf71bc640 10 monclient: build_initial_monmap 2024-03-29T10:47:48.194 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.212+0000 7fcbf71bc640 10 monclient: monmap: 2024-03-29T10:47:48.194 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:48.194 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:48.194 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:48.213087+0000 2024-03-29T10:47:48.194 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:48.213087+0000 2024-03-29T10:47:48.195 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:48.195 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:48.195 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:48.195 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:48.195 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:48.195 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:48.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.214+0000 7fcbf71bc640 1 Processor -- start 2024-03-29T10:47:48.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.214+0000 7fcbf71bc640 1 -- start start 2024-03-29T10:47:48.196 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.214+0000 7fcbf71bc640 10 monclient: init 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf01457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 10 monclient(hunting): picked mon.noname-a con 0x7fcbf013f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf4f58640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fcbf01457f0 0x7fcbf013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:54256/0 (socket says 172.21.15.94:54256) 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf4f58640 1 -- 172.21.15.94:0/3061964849 learned_addr learned my addr 172.21.15.94:0/3061964849 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf00a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00aaef0 con 0x7fcbf013f650 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf0145bd0 con 0x7fcbf00a9d70 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf0145eb0 con 0x7fcbf01457f0 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:48.198 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.215+0000 7fcbf71bc640 10 monclient(hunting): authenticate will time out at 1319.483838s 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304014356 0 0) 0x7fcbf0145eb0 con 0x7fcbf01457f0 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd8003520 con 0x7fcbf01457f0 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 261407870 0 0) 0x7fcbd8003520 con 0x7fcbf01457f0 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 >> v1:172.21.15.173:6789/0 conn(0x7fcbf00a9d70 legacy=0x7fcbf014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.216+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 >> v1:172.21.15.94:6789/0 conn(0x7fcbf013f650 legacy=0x7fcbf00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 1 monclient: found mon.noname-b 2024-03-29T10:47:48.201 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:48.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:47:48.202 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf0146340 con 0x7fcbf01457f0 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcbe8001a10 con 0x7fcbf01457f0 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.218+0000 7fcbef7fe640 10 monclient: dump: 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:48.203 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbe8001be0 con 0x7fcbf01457f0 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbef7fe640 1 -- 172.21.15.94:0/3061964849 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcbe8002030 con 0x7fcbf01457f0 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbef7fe640 10 monclient: dump: 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.204 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbf71bc640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbf71bc640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.219+0000 7fcbf71bc640 4 set_mon_vals no callback set 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.220+0000 7fcbf71bc640 10 monclient: shutdown 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.220+0000 7fcbf71bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.220+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 >> v1:172.21.15.94:6790/0 conn(0x7fcbf01457f0 legacy=0x7fcbf013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.220+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 shutdown_connections 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.220+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 >> 172.21.15.94:0/3061964849 conn(0x7fcbf0066a00 msgr2=0x7fcbf0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.220+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 shutdown_connections 2024-03-29T10:47:48.205 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.220+0000 7fcbf71bc640 1 -- 172.21.15.94:0/3061964849 wait complete. 2024-03-29T10:47:48.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.221+0000 7fcbf71bc640 10 monclient: build_initial_monmap 2024-03-29T10:47:48.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.221+0000 7fcbf71bc640 10 monclient: monmap: 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.222+0000 7fcbf71bc640 1 librados: starting msgr at 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.222+0000 7fcbf71bc640 1 librados: starting objecter 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.222+0000 7fcbf71bc640 1 Processor -- start 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.222+0000 7fcbf71bc640 1 -- start start 2024-03-29T10:47:48.211 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.222+0000 7fcbf71bc640 1 librados: setting wanted keys 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.222+0000 7fcbf71bc640 1 librados: calling monclient init 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.222+0000 7fcbf71bc640 10 monclient: init 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 10 monclient(hunting): picked mon.noname-b con 0x7fcbf00a9d70 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 10 monclient(hunting): picked mon.noname-a con 0x7fcbf013f650 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 10 monclient(hunting): picked mon.noname-c con 0x7fcbf01457f0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf0133340 con 0x7fcbf013f650 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf0130740 con 0x7fcbf00a9d70 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.223+0000 7fcbf71bc640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf0133590 con 0x7fcbf01457f0 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbf71bc640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbf71bc640 10 monclient(hunting): authenticate will time out at 1319.492180s 2024-03-29T10:47:48.212 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbf4f58640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fcbf00a9d70 0x7fcbf006f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:38738/0 (socket says 172.21.15.94:38738) 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbf4f58640 1 -- 172.21.15.94:0/1503317481 learned_addr learned my addr 172.21.15.94:0/1503317481 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbed7fa640 10 client.?.objecter ms_handle_connect 0x7fcbf00a9d70 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbed7fa640 10 client.?.objecter ms_handle_connect 0x7fcbf01457f0 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.224+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121673454 0 0) 0x7fcbf0130740 con 0x7fcbf00a9d70 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc004eb0 con 0x7fcbf00a9d70 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230352724 0 0) 0x7fcbf0133590 con 0x7fcbf01457f0 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:48.213 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf0130740 con 0x7fcbf01457f0 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304861465 0 0) 0x7fcbf0130740 con 0x7fcbf01457f0 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbf0133590 con 0x7fcbf01457f0 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcbe40017b0 con 0x7fcbf01457f0 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): dump: 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.214 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2424041 0 0) 0x7fcbdc004eb0 con 0x7fcbf00a9d70 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbf0130740 con 0x7fcbf00a9d70 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcbe8001fb0 con 0x7fcbf00a9d70 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 10 monclient(hunting): dump: 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:48.215 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1563985352 0 0) 0x7fcbf0133590 con 0x7fcbf01457f0 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 >> v1:172.21.15.173:6789/0 conn(0x7fcbf00a9d70 legacy=0x7fcbf006f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.225+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 >> v1:172.21.15.94:6789/0 conn(0x7fcbf013f650 legacy=0x7fcbf0070840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 1 monclient: found mon.c 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf013c700 con 0x7fcbf01457f0 2024-03-29T10:47:48.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbf71bc640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbf71bc640 10 monclient: _renew_subs 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbf71bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.227+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf013bc60 con 0x7fcbf01457f0 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbf71bc640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbf71bc640 10 monclient: _renew_subs 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbf71bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf01e7580 con 0x7fcbf01457f0 2024-03-29T10:47:48.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbf71bc640 1 librados: init done 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbe4001910 con 0x7fcbf01457f0 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.228+0000 7fcbeeffd640 4 set_mon_vals no callback set 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fcbe4003790 con 0x7fcbf01457f0 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbed7fa640 10 monclient: dump: 2024-03-29T10:47:48.219 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbf71bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbf71bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbf71bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.229+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd0005080 con 0x7fcbf01457f0 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.230+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fcbe4019760 con 0x7fcbf01457f0 2024-03-29T10:47:48.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.231+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fcbe40836f0 con 0x7fcbf01457f0 2024-03-29T10:47:48.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.231+0000 7fcbed7fa640 10 client.4562.objecter ms_dispatch 0x7fcbf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:48.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.231+0000 7fcbed7fa640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:48.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.231+0000 7fcbed7fa640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:48.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.231+0000 7fcbed7fa640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T10:47:48.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.232+0000 7fcbf4f58640 10 monclient: get_auth_request con 0x7fcbdc06de80 auth_method 0 2024-03-29T10:47:48.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.232+0000 7fcbed7fa640 10 client.4562.objecter ms_handle_connect 0x7fcbdc06de80 2024-03-29T10:47:48.221 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.239+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbe404efd0 con 0x7fcbf01457f0 2024-03-29T10:47:48.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.239+0000 7fcbed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:48.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.239+0000 7fcbed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:48.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.545+0000 7fcbf71bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:48.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.545+0000 7fcbf71bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:48.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.545+0000 7fcbf71bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:48.527 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.545+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbd0005330 con 0x7fcbf01457f0 2024-03-29T10:47:48.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.545+0000 7fcbed7fa640 1 -- 172.21.15.94:0/1503317481 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fcbe4050f00 con 0x7fcbf01457f0 2024-03-29T10:47:48.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.545+0000 7fcbed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:48.528 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.545+0000 7fcbed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:48.528 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:48.528 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:48.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 10 librados: watch_flush enter 2024-03-29T10:47:48.532 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 10 librados: watch_flush exit 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 10 client.4562.objecter successfully canceled tick 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 >> v1:172.21.15.94:6816/3829934808 conn(0x7fcbdc06de80 legacy=0x7fcbdc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 10 monclient: shutdown 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 >> v1:172.21.15.94:6790/0 conn(0x7fcbf01457f0 legacy=0x7fcbf0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 shutdown_connections 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 >> 172.21.15.94:0/1503317481 conn(0x7fcbf0066a00 msgr2=0x7fcbf00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 shutdown_connections 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 1 -- 172.21.15.94:0/1503317481 wait complete. 2024-03-29T10:47:48.533 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:48.549+0000 7fcbf71bc640 1 librados: shutdown 2024-03-29T10:47:48.546 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:55.353 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:55.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.501+0000 7f7926ab3640 10 monclient: get_monmap_and_config 2024-03-29T10:47:55.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.501+0000 7f7926ab3640 10 monclient: build_initial_monmap 2024-03-29T10:47:55.483 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.501+0000 7f7926ab3640 10 monclient: monmap: 2024-03-29T10:47:55.483 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:55.483 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:55.483 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:47:55.502108+0000 2024-03-29T10:47:55.483 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:47:55.502108+0000 2024-03-29T10:47:55.484 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:55.484 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:55.484 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:55.484 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:47:55.484 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:47:55.484 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:55.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f7926ab3640 1 Processor -- start 2024-03-29T10:47:55.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f7926ab3640 1 -- start start 2024-03-29T10:47:55.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f7926ab3640 10 monclient: init 2024-03-29T10:47:55.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f7926ab3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:55.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f7926ab3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:55.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f7926ab3640 10 monclient(hunting): picked mon.noname-a con 0x7f79201422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:55.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f7926ab3640 10 monclient(hunting): picked mon.noname-b con 0x7f7920142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f791ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7920142690 0x7f79200b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35924/0 (socket says 172.21.15.94:35924) 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.502+0000 7f791ffff640 1 -- 172.21.15.94:0/83175624 learned_addr learned my addr 172.21.15.94:0/83175624 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f7926ab3640 10 monclient(hunting): picked mon.noname-c con 0x7f79200b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7920142c80 con 0x7f79201422b0 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79200aae40 con 0x7f79200b17d0 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792013fbd0 con 0x7f7920142690 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f7926ab3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f7926ab3640 10 monclient(hunting): authenticate will time out at 1326.771307s 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152266326 0 0) 0x7f792013fbd0 con 0x7f7920142690 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7910003520 con 0x7f7920142690 2024-03-29T10:47:55.487 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196027145 0 0) 0x7f7910003520 con 0x7f7920142690 2024-03-29T10:47:55.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 >> v1:172.21.15.173:6789/0 conn(0x7f79200b17d0 legacy=0x7f79200b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.503+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 >> v1:172.21.15.94:6789/0 conn(0x7f79201422b0 legacy=0x7f792013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:55.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:55.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:47:55.489 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7920150930 con 0x7f7920142690 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f790c001690 con 0x7f7920142690 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f791f7fe640 10 monclient: dump: 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.490 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f7926ab3640 5 monclient: authenticate success, global_id 4565 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.506+0000 7f7926ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f790c0017f0 con 0x7f7920142690 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f791f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f791f7fe640 1 -- 172.21.15.94:0/83175624 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f790c001c40 con 0x7f7920142690 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f791f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f791f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f791f7fe640 10 monclient: dump: 2024-03-29T10:47:55.491 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f7926ab3640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:55.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.507+0000 7f7926ab3640 4 set_mon_vals no callback set 2024-03-29T10:47:55.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.508+0000 7f7926ab3640 10 monclient: shutdown 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.508+0000 7f7926ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.508+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 >> v1:172.21.15.94:6790/0 conn(0x7f7920142690 legacy=0x7f79200b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.509+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 shutdown_connections 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.509+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 >> 172.21.15.94:0/83175624 conn(0x7f7920066a00 msgr2=0x7f7920066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.509+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 shutdown_connections 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.509+0000 7f7926ab3640 1 -- 172.21.15.94:0/83175624 wait complete. 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.509+0000 7f7926ab3640 10 monclient: build_initial_monmap 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.509+0000 7f7926ab3640 10 monclient: monmap: 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:47:55.494 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.510+0000 7f7926ab3640 1 librados: starting msgr at 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.510+0000 7f7926ab3640 1 librados: starting objecter 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.510+0000 7f7926ab3640 1 Processor -- start 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.510+0000 7f7926ab3640 1 -- start start 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.510+0000 7f7926ab3640 1 librados: setting wanted keys 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.510+0000 7f7926ab3640 1 librados: calling monclient init 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.510+0000 7f7926ab3640 10 monclient: init 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.512+0000 7f7926ab3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:55.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.512+0000 7f7926ab3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 10 monclient(hunting): picked mon.noname-a con 0x7f79200b17d0 addr v1:172.21.15.94:6789/0 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 10 monclient(hunting): picked mon.noname-c con 0x7f79201422b0 addr v1:172.21.15.94:6790/0 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 10 monclient(hunting): picked mon.noname-b con 0x7f7920142690 addr v1:172.21.15.173:6789/0 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792012f6f0 con 0x7f79200b17d0 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792014cba0 con 0x7f7920142690 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791ffff640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f79201422b0 0x7f792012e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:38566/0 (socket says 172.21.15.94:38566) 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791ffff640 1 -- 172.21.15.94:0/3420551528 learned_addr learned my addr 172.21.15.94:0/3420551528 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792014dcf0 con 0x7f79201422b0 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f7926ab3640 10 monclient(hunting): authenticate will time out at 1326.781580s 2024-03-29T10:47:55.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 client.?.objecter ms_handle_connect 0x7f79201422b0 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992114180 0 0) 0x7f792014dcf0 con 0x7f79201422b0 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7904004eb0 con 0x7f79201422b0 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3058188520 0 0) 0x7f7904004eb0 con 0x7f79201422b0 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f792014dcf0 con 0x7f79201422b0 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f790c001b80 con 0x7f79201422b0 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): dump: 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:55.500 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.513+0000 7f791d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 717078000 0 0) 0x7f792014dcf0 con 0x7f79201422b0 2024-03-29T10:47:55.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 >> v1:172.21.15.173:6789/0 conn(0x7f7920142690 legacy=0x7f792012efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 >> v1:172.21.15.94:6789/0 conn(0x7f79200b17d0 legacy=0x7f79201315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 1 monclient: found mon.c 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.514+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792014dfd0 con 0x7f79201422b0 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f791d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f791d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f791d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f791d7fa640 10 client.?.objecter ms_handle_connect 0x7f79200b17d0 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f791d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 10 monclient: _renew_subs 2024-03-29T10:47:55.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792014d080 con 0x7f79201422b0 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 10 monclient: _renew_subs 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7920131ea0 con 0x7f79201422b0 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.515+0000 7f7926ab3640 1 librados: init done 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.516+0000 7f7926ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.516+0000 7f7926ab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.516+0000 7f7926ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.516+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792014d270 con 0x7f79201422b0 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.517+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f790c001ce0 con 0x7f79201422b0 2024-03-29T10:47:55.503 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.517+0000 7f791d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.517+0000 7f791effd640 4 set_mon_vals no callback set 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.517+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f790c004c30 con 0x7f79201422b0 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.518+0000 7f791d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.518+0000 7f791d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.518+0000 7f791d7fa640 10 monclient: dump: 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:55.505 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:47:55.506 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:47:55.506 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:47:55.506 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:47:55.506 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.518+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f790c01ac80 con 0x7f79201422b0 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.519+0000 7f792484f640 10 monclient: get_auth_request con 0x7f790406dbe0 auth_method 0 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.519+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f790c084c30 con 0x7f79201422b0 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.519+0000 7f791d7fa640 10 client.4568.objecter ms_dispatch 0x7f7920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.519+0000 7f791d7fa640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.519+0000 7f791d7fa640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.519+0000 7f791d7fa640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T10:47:55.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.519+0000 7f791d7fa640 10 client.4568.objecter ms_handle_connect 0x7f790406dbe0 2024-03-29T10:47:55.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.525+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f790c0503c0 con 0x7f79201422b0 2024-03-29T10:47:55.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.525+0000 7f791d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:55.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.525+0000 7f791d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:55.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.834+0000 7f7926ab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:55.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.835+0000 7f7926ab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:55.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.835+0000 7f7926ab3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:47:55.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.835+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79200b2270 con 0x7f79201422b0 2024-03-29T10:47:55.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.835+0000 7f791d7fa640 1 -- 172.21.15.94:0/3420551528 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f790c0524c0 con 0x7f79201422b0 2024-03-29T10:47:55.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.835+0000 7f791d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:55.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.835+0000 7f791d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:55.818 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:47:55.818 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 10 librados: watch_flush enter 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 10 librados: watch_flush exit 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 10 client.4568.objecter successfully canceled tick 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 >> v1:172.21.15.94:6816/3829934808 conn(0x7f790406dbe0 legacy=0x7f79040700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 10 monclient: shutdown 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 >> v1:172.21.15.94:6790/0 conn(0x7f79201422b0 legacy=0x7f792012e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 shutdown_connections 2024-03-29T10:47:55.822 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 >> 172.21.15.94:0/3420551528 conn(0x7f7920066a00 msgr2=0x7f79200b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:55.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 shutdown_connections 2024-03-29T10:47:55.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 1 -- 172.21.15.94:0/3420551528 wait complete. 2024-03-29T10:47:55.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:47:55.839+0000 7f7926ab3640 1 librados: shutdown 2024-03-29T10:47:55.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:02.643 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:02.797 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.814+0000 7fd29dc06640 10 monclient: get_monmap_and_config 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.814+0000 7fd29dc06640 10 monclient: build_initial_monmap 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.814+0000 7fd29dc06640 10 monclient: monmap: 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:02.815806+0000 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:02.815806+0000 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:02.798 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:02.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.815+0000 7fd29dc06640 1 Processor -- start 2024-03-29T10:48:02.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.816+0000 7fd29dc06640 1 -- start start 2024-03-29T10:48:02.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.816+0000 7fd29dc06640 10 monclient: init 2024-03-29T10:48:02.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.816+0000 7fd29dc06640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:02.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.816+0000 7fd29dc06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:02.799 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.816+0000 7fd29dc06640 10 monclient(hunting): picked mon.noname-b con 0x7fd2980b0770 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:02.800 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.818+0000 7fd2977fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fd2980b0770 0x7fd298149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:59888/0 (socket says 172.21.15.94:59888) 2024-03-29T10:48:02.801 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.820+0000 7fd2977fe640 1 -- 172.21.15.94:0/1159705161 learned_addr learned my addr 172.21.15.94:0/1159705161 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:02.802 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.820+0000 7fd29dc06640 10 monclient(hunting): picked mon.noname-a con 0x7fd29814a700 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:02.802 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.820+0000 7fd29dc06640 10 monclient(hunting): picked mon.noname-c con 0x7fd2980ac5a0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:02.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.820+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2980aba70 con 0x7fd29814a700 2024-03-29T10:48:02.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2980aecb0 con 0x7fd2980ac5a0 2024-03-29T10:48:02.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2980af150 con 0x7fd2980b0770 2024-03-29T10:48:02.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd29dc06640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:02.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd29dc06640 10 monclient(hunting): authenticate will time out at 1334.089242s 2024-03-29T10:48:02.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763486960 0 0) 0x7fd2980af150 con 0x7fd2980b0770 2024-03-29T10:48:02.803 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:02.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:02.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:02.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:48:02.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd280003520 con 0x7fd2980b0770 2024-03-29T10:48:02.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421988697 0 0) 0x7fd280003520 con 0x7fd2980b0770 2024-03-29T10:48:02.804 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.821+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 >> v1:172.21.15.173:6789/0 conn(0x7fd2980ac5a0 legacy=0x7fd298151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:02.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 >> v1:172.21.15.94:6789/0 conn(0x7fd29814a700 legacy=0x7fd2980abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:02.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:02.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:02.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:02.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:48:02.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980ac9d0 con 0x7fd2980b0770 2024-03-29T10:48:02.806 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd288001690 con 0x7fd2980b0770 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd2967fc640 10 monclient: dump: 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:02.807 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:02.808 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:02.808 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:02.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd29dc06640 5 monclient: authenticate success, global_id 4571 2024-03-29T10:48:02.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.823+0000 7fd29dc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:02.808 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd288001920 con 0x7fd2980b0770 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd2967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd29dc06640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd29dc06640 4 set_mon_vals no callback set 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd2967fc640 1 -- 172.21.15.94:0/1159705161 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd288001cf0 con 0x7fd2980b0770 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd2967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd2967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.825+0000 7fd2967fc640 10 monclient: dump: 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:02.809 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:02.810 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:02.810 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:02.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.826+0000 7fd29dc06640 10 monclient: shutdown 2024-03-29T10:48:02.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.826+0000 7fd29dc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:02.810 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.826+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 >> v1:172.21.15.94:6790/0 conn(0x7fd2980b0770 legacy=0x7fd298149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.827+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 shutdown_connections 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.827+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 >> 172.21.15.94:0/1159705161 conn(0x7fd29806b660 msgr2=0x7fd29806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.828+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 shutdown_connections 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.828+0000 7fd29dc06640 1 -- 172.21.15.94:0/1159705161 wait complete. 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.829+0000 7fd29dc06640 10 monclient: build_initial_monmap 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.829+0000 7fd29dc06640 10 monclient: monmap: 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:02.811 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:02.812 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:02.812 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:02.812 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:02.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.829+0000 7fd29dc06640 1 librados: starting msgr at 2024-03-29T10:48:02.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.829+0000 7fd29dc06640 1 librados: starting objecter 2024-03-29T10:48:02.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.830+0000 7fd29dc06640 1 Processor -- start 2024-03-29T10:48:02.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.830+0000 7fd29dc06640 1 -- start start 2024-03-29T10:48:02.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.830+0000 7fd29dc06640 1 librados: setting wanted keys 2024-03-29T10:48:02.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.830+0000 7fd29dc06640 1 librados: calling monclient init 2024-03-29T10:48:02.813 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.830+0000 7fd29dc06640 10 monclient: init 2024-03-29T10:48:02.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.831+0000 7fd29dc06640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:02.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.831+0000 7fd29dc06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:02.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.831+0000 7fd29dc06640 10 monclient(hunting): picked mon.noname-c con 0x7fd2980ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:02.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.831+0000 7fd2977fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fd2980ac5a0 0x7fd298126690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:33772/0 (socket says 172.21.15.94:33772) 2024-03-29T10:48:02.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.831+0000 7fd2977fe640 1 -- 172.21.15.94:0/537243225 learned_addr learned my addr 172.21.15.94:0/537243225 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:02.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.832+0000 7fd29dc06640 10 monclient(hunting): picked mon.noname-a con 0x7fd2980b0770 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:02.814 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.832+0000 7fd29dc06640 10 monclient(hunting): picked mon.noname-b con 0x7fd29814a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:02.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.832+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2981271a0 con 0x7fd2980b0770 2024-03-29T10:48:02.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.832+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd298125c20 con 0x7fd29814a700 2024-03-29T10:48:02.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.832+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd298126da0 con 0x7fd2980ac5a0 2024-03-29T10:48:02.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.832+0000 7fd29dc06640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:02.815 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.832+0000 7fd29dc06640 10 monclient(hunting): authenticate will time out at 1334.101049s 2024-03-29T10:48:02.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 10 client.?.objecter ms_handle_connect 0x7fd2980ac5a0 2024-03-29T10:48:02.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:02.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394749622 0 0) 0x7fd298126da0 con 0x7fd2980ac5a0 2024-03-29T10:48:02.817 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:02.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:02.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:02.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:48:02.818 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.833+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd284004eb0 con 0x7fd2980ac5a0 2024-03-29T10:48:02.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.836+0000 7fd27ffff640 10 client.?.objecter ms_handle_connect 0x7fd2980b0770 2024-03-29T10:48:02.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.836+0000 7fd27ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:02.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425042349 0 0) 0x7fd2981271a0 con 0x7fd2980b0770 2024-03-29T10:48:02.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:02.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:02.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:02.819 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:48:02.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd298126da0 con 0x7fd2980b0770 2024-03-29T10:48:02.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545375683 0 0) 0x7fd298126da0 con 0x7fd2980b0770 2024-03-29T10:48:02.820 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2981271a0 con 0x7fd2980b0770 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.837+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd28c0017c0 con 0x7fd2980b0770 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.838+0000 7fd27ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.838+0000 7fd27ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.838+0000 7fd27ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:02.823 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.838+0000 7fd27ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.838+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1817709565 0 0) 0x7fd284004eb0 con 0x7fd2980ac5a0 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.838+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd298126da0 con 0x7fd2980ac5a0 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.838+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3121563179 0 0) 0x7fd2981271a0 con 0x7fd2980b0770 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 >> v1:172.21.15.94:6790/0 conn(0x7fd2980ac5a0 legacy=0x7fd298126690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 >> v1:172.21.15.173:6789/0 conn(0x7fd29814a700 legacy=0x7fd2981dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 1 monclient: found mon.a 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298145a30 con 0x7fd2980b0770 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:02.824 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd27ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd29dc06640 5 monclient: authenticate success, global_id 4551 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd29dc06640 10 monclient: _renew_subs 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.839+0000 7fd29dc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd298145c80 con 0x7fd2980b0770 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd29dc06640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd29dc06640 10 monclient: _renew_subs 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd29dc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2981e7080 con 0x7fd2980b0770 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd29dc06640 1 librados: init done 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd28c001920 con 0x7fd2980b0770 2024-03-29T10:48:02.825 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd27ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:02.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.840+0000 7fd295ffb640 4 set_mon_vals no callback set 2024-03-29T10:48:02.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.842+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fd28c0037b0 con 0x7fd2980b0770 2024-03-29T10:48:02.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.842+0000 7fd27ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:02.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.842+0000 7fd27ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:02.827 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.842+0000 7fd27ffff640 10 monclient: dump: 2024-03-29T10:48:02.827 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:02.828 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:02.828 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.842+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fd28c019780 con 0x7fd2980b0770 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd29dc06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd29dc06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7fd28c0834d0 con 0x7fd2980b0770 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd27ffff640 10 client.4551.objecter ms_dispatch 0x7fd298000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd27ffff640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:02.829 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd27ffff640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:02.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd27ffff640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T10:48:02.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd29dc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:02.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.843+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd278005080 con 0x7fd2980b0770 2024-03-29T10:48:02.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.844+0000 7fd2977fe640 10 monclient: get_auth_request con 0x7fd28406dde0 auth_method 0 2024-03-29T10:48:02.830 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.844+0000 7fd27ffff640 10 client.4551.objecter ms_handle_connect 0x7fd28406dde0 2024-03-29T10:48:02.835 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.852+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd28c04edd0 con 0x7fd2980b0770 2024-03-29T10:48:02.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.853+0000 7fd27ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:02.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:02.853+0000 7fd27ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:03.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.183+0000 7fd29dc06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:03.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.183+0000 7fd29dc06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:03.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.183+0000 7fd29dc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:03.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.183+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd278005330 con 0x7fd2980b0770 2024-03-29T10:48:03.170 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:03.170 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:03.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.184+0000 7fd27ffff640 1 -- 172.21.15.94:0/537243225 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7fd28c050d00 con 0x7fd2980b0770 2024-03-29T10:48:03.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.184+0000 7fd27ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:03.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.184+0000 7fd27ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:03.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.187+0000 7fd29dc06640 10 librados: watch_flush enter 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.188+0000 7fd29dc06640 10 librados: watch_flush exit 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.188+0000 7fd29dc06640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.188+0000 7fd29dc06640 10 client.4551.objecter successfully canceled tick 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.188+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 >> v1:172.21.15.94:6816/3829934808 conn(0x7fd28406dde0 legacy=0x7fd2840702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.188+0000 7fd29dc06640 10 monclient: shutdown 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.188+0000 7fd29dc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.188+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 >> v1:172.21.15.94:6789/0 conn(0x7fd2980b0770 legacy=0x7fd2981db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.189+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 shutdown_connections 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.189+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 >> 172.21.15.94:0/537243225 conn(0x7fd29806b660 msgr2=0x7fd2980b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.189+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 shutdown_connections 2024-03-29T10:48:03.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.189+0000 7fd29dc06640 1 -- 172.21.15.94:0/537243225 wait complete. 2024-03-29T10:48:03.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:03.189+0000 7fd29dc06640 1 librados: shutdown 2024-03-29T10:48:03.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:09.997 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.168+0000 7f5ef0c33640 10 monclient: get_monmap_and_config 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.169+0000 7f5ef0c33640 10 monclient: build_initial_monmap 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.169+0000 7f5ef0c33640 10 monclient: monmap: 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:10.169890+0000 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:10.169890+0000 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:10.151 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:10.152 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:10.152 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:10.152 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:10.152 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:10.152 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.169+0000 7f5ef0c33640 1 Processor -- start 2024-03-29T10:48:10.152 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.170+0000 7f5ef0c33640 1 -- start start 2024-03-29T10:48:10.152 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.170+0000 7f5ef0c33640 10 monclient: init 2024-03-29T10:48:10.153 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:10.156 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:10.156 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 10 monclient(hunting): picked mon.noname-a con 0x7f5eec1422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:10.156 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 10 monclient(hunting): picked mon.noname-b con 0x7f5eec142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:10.156 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 10 monclient(hunting): picked mon.noname-c con 0x7f5eec0b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:10.156 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec142c80 con 0x7f5eec1422b0 2024-03-29T10:48:10.157 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec0aae40 con 0x7f5eec0b17d0 2024-03-29T10:48:10.157 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec13fbd0 con 0x7f5eec142690 2024-03-29T10:48:10.157 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:10.157 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.171+0000 7f5ef0c33640 10 monclient(hunting): authenticate will time out at 1341.439313s 2024-03-29T10:48:10.157 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.174+0000 7f5eead9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f5eec0b17d0 0x7f5eec0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:51009/0 (socket says 172.21.15.94:51009) 2024-03-29T10:48:10.157 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.174+0000 7f5eead9d640 1 -- 172.21.15.94:0/1081565608 learned_addr learned my addr 172.21.15.94:0/1081565608 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218850773 0 0) 0x7f5eec0aae40 con 0x7f5eec0b17d0 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 10 monclient(hunting): _negotiate 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed4003520 con 0x7f5eec0b17d0 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533915895 0 0) 0x7f5ed4003520 con 0x7f5eec0b17d0 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 >> v1:172.21.15.94:6790/0 conn(0x7f5eec142690 legacy=0x7f5eec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.176+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 >> v1:172.21.15.94:6789/0 conn(0x7f5eec1422b0 legacy=0x7f5eec13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:10.159 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: _finish_hunting 0 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 1 monclient: found mon.noname-c 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec151940 con 0x7f5eec0b17d0 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: _finish_auth 0 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: _check_auth_tickets 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5ee0001690 con 0x7f5eec0b17d0 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: dump: 2024-03-29T10:48:10.160 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ef0c33640 5 monclient: authenticate success, global_id 4513 2024-03-29T10:48:10.161 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ef0c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ee00018b0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 1 -- 172.21.15.94:0/1081565608 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5ee0001cc0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ee959a640 10 monclient: dump: 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:10.163 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ef0c33640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.177+0000 7f5ef0c33640 4 set_mon_vals no callback set 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.178+0000 7f5ef0c33640 10 monclient: shutdown 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.178+0000 7f5ef0c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.178+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1081565608 >> v1:172.21.15.173:6789/0 conn(0x7f5eec0b17d0 legacy=0x7f5eec0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.178+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1081565608 shutdown_connections 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.179+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1081565608 >> 172.21.15.94:0/1081565608 conn(0x7f5eec066a00 msgr2=0x7f5eec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.179+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1081565608 shutdown_connections 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.179+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1081565608 wait complete. 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 10 monclient: build_initial_monmap 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 10 monclient: monmap: 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:10.164 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 1 librados: starting msgr at 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 1 librados: starting objecter 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 1 Processor -- start 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 1 -- start start 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 1 librados: setting wanted keys 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 1 librados: calling monclient init 2024-03-29T10:48:10.165 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.180+0000 7f5ef0c33640 10 monclient: init 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 10 monclient(hunting): picked mon.noname-c con 0x7f5eec0b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 10 monclient(hunting): picked mon.noname-b con 0x7f5eec1422b0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 10 monclient(hunting): picked mon.noname-a con 0x7f5eec142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec06e930 con 0x7f5eec142690 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec06eb80 con 0x7f5eec1422b0 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec0b74e0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:10.166 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5ef0c33640 10 monclient(hunting): authenticate will time out at 1341.449689s 2024-03-29T10:48:10.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5eea59c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f5eec0b17d0 0x7f5eec0b6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:37802/0 (socket says 172.21.15.94:37802) 2024-03-29T10:48:10.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.181+0000 7f5eea59c640 1 -- 172.21.15.94:0/1900604249 learned_addr learned my addr 172.21.15.94:0/1900604249 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:10.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 10 client.?.objecter ms_handle_connect 0x7f5eec0b17d0 2024-03-29T10:48:10.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:10.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925444384 0 0) 0x7f5eec0b74e0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.169 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.182+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ecc004eb0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3033197748 0 0) 0x7f5ecc004eb0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ecc005300 con 0x7f5eec0b17d0 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5ee4001ca0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 10 monclient(hunting): dump: 2024-03-29T10:48:10.170 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1766288237 0 0) 0x7f5ecc005300 con 0x7f5eec0b17d0 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.183+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 >> v1:172.21.15.173:6789/0 conn(0x7f5eec1422b0 legacy=0x7f5eec12ebb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 >> v1:172.21.15.94:6789/0 conn(0x7f5eec142690 legacy=0x7f5eec1db9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:10.171 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 1 monclient: found mon.c 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec0b64e0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5edaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 5 monclient: authenticate success, global_id 4580 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 10 monclient: _renew_subs 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec0b6730 con 0x7f5eec0b17d0 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 10 monclient: _renew_subs 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eec12f4a0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.184+0000 7f5ef0c33640 1 librados: init done 2024-03-29T10:48:10.172 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.186+0000 7f5ef0c33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.186+0000 7f5ef0c33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.186+0000 7f5ef0c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.186+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eec0b7320 con 0x7f5eec0b17d0 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.187+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ee4001fc0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.187+0000 7f5edaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.187+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5ee4001ca0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.187+0000 7f5edaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.187+0000 7f5edaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.187+0000 7f5edaffd640 10 monclient: dump: 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:10.173 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:10.174 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:10.174 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:10.174 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:10.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.187+0000 7f5edbfff640 4 set_mon_vals no callback set 2024-03-29T10:48:10.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.188+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f5ee4003ca0 con 0x7f5eec0b17d0 2024-03-29T10:48:10.174 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.189+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f5ee4054770 con 0x7f5eec0b17d0 2024-03-29T10:48:10.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.189+0000 7f5edaffd640 10 client.4580.objecter ms_dispatch 0x7f5eec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:10.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.189+0000 7f5edaffd640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:10.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.189+0000 7f5edaffd640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:10.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.189+0000 7f5edaffd640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T10:48:10.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.189+0000 7f5ee9d9b640 10 monclient: get_auth_request con 0x7f5ecc06e1d0 auth_method 0 2024-03-29T10:48:10.175 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.189+0000 7f5edaffd640 10 client.4580.objecter ms_handle_connect 0x7f5ecc06e1d0 2024-03-29T10:48:10.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.195+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ee4050a10 con 0x7f5eec0b17d0 2024-03-29T10:48:10.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.195+0000 7f5edaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:10.177 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.195+0000 7f5edaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:10.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.512+0000 7f5ef0c33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:10.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.512+0000 7f5ef0c33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:10.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.512+0000 7f5ef0c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:10.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.512+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eec075f10 con 0x7f5eec0b17d0 2024-03-29T10:48:10.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.513+0000 7f5edaffd640 1 -- 172.21.15.94:0/1900604249 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f5ee4050a10 con 0x7f5eec0b17d0 2024-03-29T10:48:10.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.513+0000 7f5edaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:10.495 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.513+0000 7f5edaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:10.496 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:10.496 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:10.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 10 librados: watch_flush enter 2024-03-29T10:48:10.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 10 librados: watch_flush exit 2024-03-29T10:48:10.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T10:48:10.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 10 client.4580.objecter successfully canceled tick 2024-03-29T10:48:10.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 >> v1:172.21.15.94:6816/3829934808 conn(0x7f5ecc06e1d0 legacy=0x7f5ecc070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 10 monclient: shutdown 2024-03-29T10:48:10.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:10.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 >> v1:172.21.15.94:6790/0 conn(0x7f5eec0b17d0 legacy=0x7f5eec0b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:10.500 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 shutdown_connections 2024-03-29T10:48:10.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 >> 172.21.15.94:0/1900604249 conn(0x7f5eec066a00 msgr2=0x7f5eec13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:10.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.517+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 shutdown_connections 2024-03-29T10:48:10.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.518+0000 7f5ef0c33640 1 -- 172.21.15.94:0/1900604249 wait complete. 2024-03-29T10:48:10.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:10.518+0000 7f5ef0c33640 1 librados: shutdown 2024-03-29T10:48:10.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:17.324 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:17.479 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.495+0000 7f04cb872640 10 monclient: get_monmap_and_config 2024-03-29T10:48:17.479 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.495+0000 7f04cb872640 10 monclient: build_initial_monmap 2024-03-29T10:48:17.479 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.495+0000 7f04cb872640 10 monclient: monmap: 2024-03-29T10:48:17.479 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:17.496353+0000 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:17.496353+0000 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:17.480 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:17.481 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.498+0000 7f04cb872640 1 Processor -- start 2024-03-29T10:48:17.484 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.501+0000 7f04cb872640 1 -- start start 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.501+0000 7f04cb872640 10 monclient: init 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 10 monclient(hunting): picked mon.noname-c con 0x7f04c40a9ac0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 10 monclient(hunting): picked mon.noname-b con 0x7f04c413ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 10 monclient(hunting): picked mon.noname-a con 0x7f04c40af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c40afd10 con 0x7f04c40af900 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c40af6f0 con 0x7f04c40a9ac0 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c40aac80 con 0x7f04c413ee90 2024-03-29T10:48:17.485 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:17.486 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.502+0000 7f04cb872640 10 monclient(hunting): authenticate will time out at 1348.770335s 2024-03-29T10:48:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.505+0000 7f04c9e0f640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f04c40af900 0x7f04c4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:38063/0 (socket says 172.21.15.94:38063) 2024-03-29T10:48:17.488 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.505+0000 7f04c9e0f640 1 -- 172.21.15.94:0/3292877367 learned_addr learned my addr 172.21.15.94:0/3292877367 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369863717 0 0) 0x7f04c40afd10 con 0x7f04c40af900 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b0003520 con 0x7f04c40af900 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447877704 0 0) 0x7f04b0003520 con 0x7f04c40af900 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 >> v1:172.21.15.94:6790/0 conn(0x7f04c413ee90 legacy=0x7f04c413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.490 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.507+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 >> v1:172.21.15.173:6789/0 conn(0x7f04c40a9ac0 legacy=0x7f04c40adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:17.491 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 1 monclient: found mon.noname-a 2024-03-29T10:48:17.492 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c40aaf60 con 0x7f04c40af900 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 10 monclient: _finish_auth 0 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04cb872640 5 monclient: authenticate success, global_id 4554 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04cb872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f04b4001690 con 0x7f04c40af900 2024-03-29T10:48:17.493 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04bbfff640 10 monclient: dump: 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:17.494 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.509+0000 7f04cb872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:17.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04b4001860 con 0x7f04c40af900 2024-03-29T10:48:17.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:17.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04bbfff640 1 -- 172.21.15.94:0/3292877367 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f04b4001cb0 con 0x7f04c40af900 2024-03-29T10:48:17.499 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.501 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04bbfff640 10 monclient: dump: 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04cb872640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:17.502 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.513+0000 7f04cb872640 4 set_mon_vals no callback set 2024-03-29T10:48:17.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.514+0000 7f04cb872640 10 monclient: shutdown 2024-03-29T10:48:17.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.514+0000 7f04cb872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:17.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.514+0000 7f04cb872640 1 -- 172.21.15.94:0/3292877367 >> v1:172.21.15.94:6789/0 conn(0x7f04c40af900 legacy=0x7f04c4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.505 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.514+0000 7f04cb872640 1 -- 172.21.15.94:0/3292877367 shutdown_connections 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.514+0000 7f04cb872640 1 -- 172.21.15.94:0/3292877367 >> 172.21.15.94:0/3292877367 conn(0x7f04c40b35e0 msgr2=0x7f04c40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.514+0000 7f04cb872640 1 -- 172.21.15.94:0/3292877367 shutdown_connections 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.514+0000 7f04cb872640 1 -- 172.21.15.94:0/3292877367 wait complete. 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.515+0000 7f04cb872640 10 monclient: build_initial_monmap 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.515+0000 7f04cb872640 10 monclient: monmap: 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:17.506 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.516+0000 7f04cb872640 1 librados: starting msgr at 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.516+0000 7f04cb872640 1 librados: starting objecter 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.516+0000 7f04cb872640 1 Processor -- start 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.516+0000 7f04cb872640 1 -- start start 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.516+0000 7f04cb872640 1 librados: setting wanted keys 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.516+0000 7f04cb872640 1 librados: calling monclient init 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.516+0000 7f04cb872640 10 monclient: init 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 10 monclient(hunting): picked mon.noname-b con 0x7f04c40a9ac0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 10 monclient(hunting): picked mon.noname-a con 0x7f04c40af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 10 monclient(hunting): picked mon.noname-c con 0x7f04c413ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:17.507 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c4140ee0 con 0x7f04c40af900 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c4141f40 con 0x7f04c40a9ac0 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c4142220 con 0x7f04c413ee90 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.517+0000 7f04cb872640 10 monclient(hunting): authenticate will time out at 1348.785514s 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.521+0000 7f04c9e0f640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f04c413ee90 0x7f04c41e6df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:35517/0 (socket says 172.21.15.94:35517) 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.521+0000 7f04c9e0f640 1 -- 172.21.15.94:0/659568990 learned_addr learned my addr 172.21.15.94:0/659568990 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.521+0000 7f04b9ffb640 10 client.?.objecter ms_handle_connect 0x7f04c413ee90 2024-03-29T10:48:17.508 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.521+0000 7f04b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:17.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 client.?.objecter ms_handle_connect 0x7f04c40af900 2024-03-29T10:48:17.509 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 client.?.objecter ms_handle_connect 0x7f04c40a9ac0 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151184106 0 0) 0x7f04c4142220 con 0x7f04c413ee90 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a8004eb0 con 0x7f04c413ee90 2024-03-29T10:48:17.510 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767186308 0 0) 0x7f04c4140ee0 con 0x7f04c40af900 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.525+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c4142220 con 0x7f04c40af900 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691997980 0 0) 0x7f04c4142220 con 0x7f04c40af900 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a8005490 con 0x7f04c40af900 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f04bc001b90 con 0x7f04c40af900 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): dump: 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:17.511 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420589092 0 0) 0x7f04c4141f40 con 0x7f04c40a9ac0 2024-03-29T10:48:17.512 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a80059c0 con 0x7f04c40a9ac0 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346768317 0 0) 0x7f04a8004eb0 con 0x7f04c413ee90 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04c4141f40 con 0x7f04c413ee90 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f04b4001b70 con 0x7f04c413ee90 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient(hunting): dump: 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.513 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2131900661 0 0) 0x7f04a8005490 con 0x7f04c40af900 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 >> v1:172.21.15.94:6790/0 conn(0x7f04c413ee90 legacy=0x7f04c41e6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 >> v1:172.21.15.173:6789/0 conn(0x7f04c40a9ac0 legacy=0x7f04c4136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 monclient: found mon.a 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:17.514 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c4141370 con 0x7f04c40af900 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04cb872640 5 monclient: authenticate success, global_id 4557 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04cb872640 10 monclient: _renew_subs 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04cb872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.526+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c4136aa0 con 0x7f04c40af900 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.527+0000 7f04cb872640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.527+0000 7f04cb872640 10 monclient: _renew_subs 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.527+0000 7f04cb872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.527+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c41402f0 con 0x7f04c40af900 2024-03-29T10:48:17.515 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.527+0000 7f04cb872640 1 librados: init done 2024-03-29T10:48:17.517 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.530+0000 7f04cb872640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.530+0000 7f04cb872640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.530+0000 7f04cb872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.530+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c4075f10 con 0x7f04c40af900 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.530+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04bc002040 con 0x7f04c40af900 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.530+0000 7f04b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.530+0000 7f04bb7fe640 4 set_mon_vals no callback set 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.531+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f04bc001b90 con 0x7f04c40af900 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.531+0000 7f04b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.531+0000 7f04b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.531+0000 7f04b9ffb640 10 monclient: dump: 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:17.518 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.532+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f04bc0197e0 con 0x7f04c40af900 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.533+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f04bc053220 con 0x7f04c40af900 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.533+0000 7f04b9ffb640 10 client.4557.objecter ms_dispatch 0x7f04c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.533+0000 7f04b9ffb640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.533+0000 7f04b9ffb640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.533+0000 7f04b9ffb640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-29T10:48:17.519 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.533+0000 7f04c960e640 10 monclient: get_auth_request con 0x7f04a806e610 auth_method 0 2024-03-29T10:48:17.520 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.534+0000 7f04b9ffb640 10 client.4557.objecter ms_handle_connect 0x7f04a806e610 2024-03-29T10:48:17.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.542+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04bc04f4c0 con 0x7f04c40af900 2024-03-29T10:48:17.524 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.542+0000 7f04b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:17.525 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.542+0000 7f04b9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:17.846 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.864+0000 7f04cb872640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:17.846 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.864+0000 7f04cb872640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:17.846 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.864+0000 7f04cb872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.94:6789/0 2024-03-29T10:48:17.846 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.864+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 --> v1:172.21.15.94:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04c4141500 con 0x7f04c40af900 2024-03-29T10:48:17.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.867+0000 7f04b9ffb640 1 -- 172.21.15.94:0/659568990 <== mon.0 v1:172.21.15.94:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f04bc04f4c0 con 0x7f04c40af900 2024-03-29T10:48:17.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.867+0000 7f04b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:17.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.868+0000 7f04b9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:17.850 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:17.850 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:17.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.871+0000 7f04cb872640 10 librados: watch_flush enter 2024-03-29T10:48:17.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.871+0000 7f04cb872640 10 librados: watch_flush exit 2024-03-29T10:48:17.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.871+0000 7f04cb872640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-29T10:48:17.854 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.871+0000 7f04cb872640 10 client.4557.objecter successfully canceled tick 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 >> v1:172.21.15.94:6816/3829934808 conn(0x7f04a806e610 legacy=0x7f04a8070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 10 monclient: shutdown 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 >> v1:172.21.15.94:6789/0 conn(0x7f04c40af900 legacy=0x7f04c413c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 shutdown_connections 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 >> 172.21.15.94:0/659568990 conn(0x7f04c40b35e0 msgr2=0x7f04c40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 shutdown_connections 2024-03-29T10:48:17.856 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 1 -- 172.21.15.94:0/659568990 wait complete. 2024-03-29T10:48:17.857 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:17.874+0000 7f04cb872640 1 librados: shutdown 2024-03-29T10:48:17.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:24.683 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.854+0000 7f7b9d4c5640 10 monclient: get_monmap_and_config 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.854+0000 7f7b9d4c5640 10 monclient: build_initial_monmap 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.854+0000 7f7b9d4c5640 10 monclient: monmap: 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:24.854947+0000 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:24.854947+0000 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:24.836 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:24.837 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:24.837 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:24.837 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:24.837 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:24.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.855+0000 7f7b9d4c5640 1 Processor -- start 2024-03-29T10:48:24.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.856+0000 7f7b9d4c5640 1 -- start start 2024-03-29T10:48:24.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.856+0000 7f7b9d4c5640 10 monclient: init 2024-03-29T10:48:24.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:24.839 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 10 monclient(hunting): picked mon.noname-c con 0x7f7b90006ad0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 10 monclient(hunting): picked mon.noname-a con 0x7f7b90036870 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 10 monclient(hunting): picked mon.noname-b con 0x7f7b90036d00 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b90037150 con 0x7f7b90036870 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b90033930 con 0x7f7b90006ad0 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b90033c10 con 0x7f7b90036d00 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:24.840 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.857+0000 7f7b9d4c5640 10 monclient(hunting): authenticate will time out at 1356.125218s 2024-03-29T10:48:24.846 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b9ccc4640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7b90036d00 0x7f7b900d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:55207/0 (socket says 172.21.15.94:55207) 2024-03-29T10:48:24.846 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b9ccc4640 1 -- 172.21.15.94:0/1834005131 learned_addr learned my addr 172.21.15.94:0/1834005131 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892282848 0 0) 0x7f7b90033c10 con 0x7f7b90036d00 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88003520 con 0x7f7b90036d00 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664269574 0 0) 0x7f7b90037150 con 0x7f7b90036870 2024-03-29T10:48:24.847 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.864+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b90033c10 con 0x7f7b90036870 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2327653315 0 0) 0x7f7b88003520 con 0x7f7b90036d00 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 >> v1:172.21.15.173:6789/0 conn(0x7f7b90006ad0 legacy=0x7f7b900322c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 >> v1:172.21.15.94:6789/0 conn(0x7f7b90036870 legacy=0x7f7b900d4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 monclient: found mon.noname-b 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b900308c0 con 0x7f7b90036d00 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: _finish_auth 0 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:24.848 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7b84001690 con 0x7f7b90036d00 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: dump: 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:24.849 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b9d4c5640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b9d4c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b840017f0 con 0x7f7b90036d00 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 1 -- 172.21.15.94:0/1834005131 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7b84001c40 con 0x7f7b90036d00 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.865+0000 7f7b96ffd640 10 monclient: dump: 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:24.850 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:24.851 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:24.851 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:24.851 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:24.851 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.869+0000 7f7b9d4c5640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:24.851 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.869+0000 7f7b9d4c5640 4 set_mon_vals no callback set 2024-03-29T10:48:24.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.870+0000 7f7b9d4c5640 10 monclient: shutdown 2024-03-29T10:48:24.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.870+0000 7f7b9d4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:24.852 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.870+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1834005131 >> v1:172.21.15.94:6790/0 conn(0x7f7b90036d00 legacy=0x7f7b900d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.870+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1834005131 shutdown_connections 2024-03-29T10:48:24.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.870+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1834005131 >> 172.21.15.94:0/1834005131 conn(0x7f7b900058f0 msgr2=0x7f7b90005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:24.853 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.872+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1834005131 shutdown_connections 2024-03-29T10:48:24.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.875+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1834005131 wait complete. 2024-03-29T10:48:24.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.875+0000 7f7b9d4c5640 10 monclient: build_initial_monmap 2024-03-29T10:48:24.858 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.875+0000 7f7b9d4c5640 10 monclient: monmap: 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.876+0000 7f7b9d4c5640 1 librados: starting msgr at 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.876+0000 7f7b9d4c5640 1 librados: starting objecter 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.876+0000 7f7b9d4c5640 1 Processor -- start 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.876+0000 7f7b9d4c5640 1 -- start start 2024-03-29T10:48:24.859 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.876+0000 7f7b9d4c5640 1 librados: setting wanted keys 2024-03-29T10:48:24.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.876+0000 7f7b9d4c5640 1 librados: calling monclient init 2024-03-29T10:48:24.860 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.876+0000 7f7b9d4c5640 10 monclient: init 2024-03-29T10:48:24.861 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 10 monclient(hunting): picked mon.noname-b con 0x7f7b90006ad0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 10 monclient(hunting): picked mon.noname-c con 0x7f7b90036870 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 10 monclient(hunting): picked mon.noname-a con 0x7f7b90036d00 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900cbeb0 con 0x7f7b90036d00 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900cb610 con 0x7f7b90006ad0 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9000d790 con 0x7f7b90036870 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:24.862 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.877+0000 7f7b9d4c5640 10 monclient(hunting): authenticate will time out at 1356.145725s 2024-03-29T10:48:24.872 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.883+0000 7f7b977fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f7b90036870 0x7f7b9000c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:55333/0 (socket says 172.21.15.94:55333) 2024-03-29T10:48:24.872 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.883+0000 7f7b977fe640 1 -- 172.21.15.94:0/1250950648 learned_addr learned my addr 172.21.15.94:0/1250950648 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.883+0000 7f7b94ff9640 10 client.?.objecter ms_handle_connect 0x7f7b90036d00 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.883+0000 7f7b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873275747 0 0) 0x7f7b900cbeb0 con 0x7f7b90036d00 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c004eb0 con 0x7f7b90036d00 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 10 client.?.objecter ms_handle_connect 0x7f7b90036870 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.884+0000 7f7b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985889041 0 0) 0x7f7b9000d790 con 0x7f7b90036870 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:24.873 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c0051d0 con 0x7f7b90036870 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 client.?.objecter ms_handle_connect 0x7f7b90006ad0 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3466454331 0 0) 0x7f7b7c0051d0 con 0x7f7b90036870 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b7c005640 con 0x7f7b90036870 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7b9804fdd0 con 0x7f7b90036870 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): dump: 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.874 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861196808 0 0) 0x7f7b900cb610 con 0x7f7b90006ad0 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.885+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c005b90 con 0x7f7b90006ad0 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796637747 0 0) 0x7f7b7c005640 con 0x7f7b90036870 2024-03-29T10:48:24.875 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 >> v1:172.21.15.173:6789/0 conn(0x7f7b90006ad0 legacy=0x7f7b901860b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 >> v1:172.21.15.94:6789/0 conn(0x7f7b90036d00 legacy=0x7f7b9000d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 1 monclient: found mon.c 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9000d920 con 0x7f7b90036870 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 10 monclient: _finish_auth 0 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 5 monclient: authenticate success, global_id 4592 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 10 monclient: _renew_subs 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b900cc3d0 con 0x7f7b90036870 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:24.876 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 10 monclient: _renew_subs 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b901869a0 con 0x7f7b90036870 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.886+0000 7f7b9d4c5640 1 librados: init done 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.887+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b9804f400 con 0x7f7b90036870 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.887+0000 7f7b94ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.887+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f7b9804fdd0 con 0x7f7b90036870 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.887+0000 7f7b94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.887+0000 7f7b94ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.887+0000 7f7b967fc640 4 set_mon_vals no callback set 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.887+0000 7f7b94ff9640 10 monclient: dump: 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:24.877 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.888+0000 7f7b9d4c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.888+0000 7f7b9d4c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.888+0000 7f7b9d4c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.888+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90004cd0 con 0x7f7b90036870 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.889+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f7b98075710 con 0x7f7b90036870 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.890+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f7b980bdf50 con 0x7f7b90036870 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.890+0000 7f7b94ff9640 10 client.4592.objecter ms_dispatch 0x7f7b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.890+0000 7f7b94ff9640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:24.878 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.890+0000 7f7b94ff9640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:24.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.890+0000 7f7b94ff9640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T10:48:24.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.890+0000 7f7b97fff640 10 monclient: get_auth_request con 0x7f7b7c06e750 auth_method 0 2024-03-29T10:48:24.879 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.891+0000 7f7b94ff9640 10 client.4592.objecter ms_handle_connect 0x7f7b7c06e750 2024-03-29T10:48:24.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.898+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b980ba1f0 con 0x7f7b90036870 2024-03-29T10:48:24.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.898+0000 7f7b94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:24.880 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:24.898+0000 7f7b94ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:25.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.227+0000 7f7b9d4c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:25.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.227+0000 7f7b9d4c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:25.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.227+0000 7f7b9d4c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:25.209 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.227+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b9000db20 con 0x7f7b90036870 2024-03-29T10:48:25.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.228+0000 7f7b94ff9640 1 -- 172.21.15.94:0/1250950648 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f7b980ba390 con 0x7f7b90036870 2024-03-29T10:48:25.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.228+0000 7f7b94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:25.210 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.228+0000 7f7b94ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:25.213 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:25.213 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 10 librados: watch_flush enter 2024-03-29T10:48:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 10 librados: watch_flush exit 2024-03-29T10:48:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T10:48:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 10 client.4592.objecter successfully canceled tick 2024-03-29T10:48:25.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 >> v1:172.21.15.94:6816/3829934808 conn(0x7f7b7c06e750 legacy=0x7f7b7c070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:25.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 10 monclient: shutdown 2024-03-29T10:48:25.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:25.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.234+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 >> v1:172.21.15.94:6790/0 conn(0x7f7b90036870 legacy=0x7f7b9000c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:25.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.235+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 shutdown_connections 2024-03-29T10:48:25.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.235+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 >> 172.21.15.94:0/1250950648 conn(0x7f7b900058f0 msgr2=0x7f7b900c7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:25.217 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.235+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 shutdown_connections 2024-03-29T10:48:25.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.238+0000 7f7b9d4c5640 1 -- 172.21.15.94:0/1250950648 wait complete. 2024-03-29T10:48:25.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:25.238+0000 7f7b9d4c5640 1 librados: shutdown 2024-03-29T10:48:25.236 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:32.043 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:32.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.231+0000 7f77b36a1640 10 monclient: get_monmap_and_config 2024-03-29T10:48:32.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.231+0000 7f77b36a1640 10 monclient: build_initial_monmap 2024-03-29T10:48:32.214 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.231+0000 7f77b36a1640 10 monclient: monmap: 2024-03-29T10:48:32.214 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:32.214 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:32.214 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:32.232730+0000 2024-03-29T10:48:32.215 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:32.232730+0000 2024-03-29T10:48:32.215 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:32.215 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:32.215 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:32.215 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:32.215 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:32.215 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:32.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.234+0000 7f77b36a1640 1 Processor -- start 2024-03-29T10:48:32.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.234+0000 7f77b36a1640 1 -- start start 2024-03-29T10:48:32.216 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.234+0000 7f77b36a1640 10 monclient: init 2024-03-29T10:48:32.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:32.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:32.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 10 monclient(hunting): picked mon.noname-c con 0x7f77ac0b0860 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:32.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 10 monclient(hunting): picked mon.noname-b con 0x7f77ac13ef30 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:32.218 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 10 monclient(hunting): picked mon.noname-a con 0x7f77ac0af8e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:32.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac0b0c40 con 0x7f77ac0af8e0 2024-03-29T10:48:32.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac0aed10 con 0x7f77ac0b0860 2024-03-29T10:48:32.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac0afd50 con 0x7f77ac13ef30 2024-03-29T10:48:32.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:32.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.235+0000 7f77b36a1640 10 monclient(hunting): authenticate will time out at 1363.503620s 2024-03-29T10:48:32.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.237+0000 7f77b143d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f77ac0b0860 0x7f77ac13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:51034/0 (socket says 172.21.15.94:51034) 2024-03-29T10:48:32.219 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.237+0000 7f77b143d640 1 -- 172.21.15.94:0/2604471640 learned_addr learned my addr 172.21.15.94:0/2604471640 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080888103 0 0) 0x7f77ac0aed10 con 0x7f77ac0b0860 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 10 monclient(hunting): my global_id is 4525 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7798003520 con 0x7f77ac0b0860 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684817612 0 0) 0x7f7798003520 con 0x7f77ac0b0860 2024-03-29T10:48:32.220 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.238+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 >> v1:172.21.15.94:6790/0 conn(0x7f77ac13ef30 legacy=0x7f77ac0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:32.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 >> v1:172.21.15.94:6789/0 conn(0x7f77ac0af8e0 legacy=0x7f77ac149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:32.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:32.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 1 monclient: found mon.noname-c 2024-03-29T10:48:32.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:32.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:48:32.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac0ab1b0 con 0x7f77ac0b0860 2024-03-29T10:48:32.222 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 10 monclient: _finish_auth 0 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77b36a1640 5 monclient: authenticate success, global_id 4525 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.239+0000 7f77b36a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.240+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f77a8001690 con 0x7f77ac0b0860 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.240+0000 7f77a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.240+0000 7f77a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.240+0000 7f77a3fff640 10 monclient: dump: 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:32.223 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.240+0000 7f77b36a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.241+0000 7f77a3fff640 1 -- 172.21.15.94:0/2604471640 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77a8001860 con 0x7f77ac0b0860 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.241+0000 7f77a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.241+0000 7f77b36a1640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:32.224 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.241+0000 7f77b36a1640 4 set_mon_vals no callback set 2024-03-29T10:48:32.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.242+0000 7f77b36a1640 10 monclient: shutdown 2024-03-29T10:48:32.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.242+0000 7f77b36a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:32.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.242+0000 7f77b36a1640 1 -- 172.21.15.94:0/2604471640 >> v1:172.21.15.173:6789/0 conn(0x7f77ac0b0860 legacy=0x7f77ac13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.243+0000 7f77b36a1640 1 -- 172.21.15.94:0/2604471640 shutdown_connections 2024-03-29T10:48:32.225 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.243+0000 7f77b36a1640 1 -- 172.21.15.94:0/2604471640 >> 172.21.15.94:0/2604471640 conn(0x7f77ac06b660 msgr2=0x7f77ac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:32.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.244+0000 7f77b36a1640 1 -- 172.21.15.94:0/2604471640 shutdown_connections 2024-03-29T10:48:32.226 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.244+0000 7f77b36a1640 1 -- 172.21.15.94:0/2604471640 wait complete. 2024-03-29T10:48:32.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.245+0000 7f77b36a1640 10 monclient: build_initial_monmap 2024-03-29T10:48:32.229 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.245+0000 7f77b36a1640 10 monclient: monmap: 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.245+0000 7f77b36a1640 1 librados: starting msgr at 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.245+0000 7f77b36a1640 1 librados: starting objecter 2024-03-29T10:48:32.230 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.245+0000 7f77b36a1640 1 Processor -- start 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 1 -- start start 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 1 librados: setting wanted keys 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 1 librados: calling monclient init 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient: init 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient(hunting): picked mon.noname-a con 0x7f77ac0af8e0 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient(hunting): picked mon.noname-c con 0x7f77ac0b0860 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient(hunting): picked mon.noname-b con 0x7f77ac13ef30 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac0705f0 con 0x7f77ac0af8e0 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac1de940 con 0x7f77ac13ef30 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac06f3b0 con 0x7f77ac0b0860 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:32.231 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.246+0000 7f77b36a1640 10 monclient(hunting): authenticate will time out at 1363.514985s 2024-03-29T10:48:32.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.250+0000 7f77b0c3c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f77ac0b0860 0x7f77ac1dee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:33575/0 (socket says 172.21.15.94:33575) 2024-03-29T10:48:32.232 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.250+0000 7f77b0c3c640 1 -- 172.21.15.94:0/2435189268 learned_addr learned my addr 172.21.15.94:0/2435189268 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:32.235 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 10 client.?.objecter ms_handle_connect 0x7f77ac13ef30 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494208237 0 0) 0x7f77ac1de940 con 0x7f77ac13ef30 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.252+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7790004eb0 con 0x7f77ac13ef30 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 client.?.objecter ms_handle_connect 0x7f77ac0b0860 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 402334101 0 0) 0x7f7790004eb0 con 0x7f77ac13ef30 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77ac1de940 con 0x7f77ac13ef30 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f779c0017f0 con 0x7f77ac13ef30 2024-03-29T10:48:32.236 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient(hunting): dump: 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3110943004 0 0) 0x7f77ac1de940 con 0x7f77ac13ef30 2024-03-29T10:48:32.237 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 >> v1:172.21.15.94:6790/0 conn(0x7f77ac0b0860 legacy=0x7f77ac1dee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 >> v1:172.21.15.94:6789/0 conn(0x7f77ac0af8e0 legacy=0x7f77ac06eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 monclient: found mon.b 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac06f540 con 0x7f77ac13ef30 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:32.238 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:32.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77b36a1640 5 monclient: authenticate success, global_id 4528 2024-03-29T10:48:32.239 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77b36a1640 10 monclient: _renew_subs 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77b36a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.253+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac1df630 con 0x7f77ac13ef30 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.254+0000 7f77b36a1640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.254+0000 7f77b36a1640 10 monclient: _renew_subs 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.254+0000 7f77b36a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.254+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac070bb0 con 0x7f77ac13ef30 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.254+0000 7f77b36a1640 1 librados: init done 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77a1ffb640 10 client.4528.objecter ms_handle_connect 0x7f77ac0af8e0 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77a1ffb640 10 client.4528.objecter resend_mon_ops 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77b36a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77b36a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77b36a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f779c0035f0 con 0x7f77ac13ef30 2024-03-29T10:48:32.240 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac075f10 con 0x7f77ac13ef30 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.255+0000 7f77a37fe640 4 set_mon_vals no callback set 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.256+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f779c001d80 con 0x7f77ac13ef30 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.256+0000 7f77a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.256+0000 7f77a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.256+0000 7f77a1ffb640 10 monclient: dump: 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:32.241 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:32.242 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:32.242 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.258+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f779c0039c0 con 0x7f77ac13ef30 2024-03-29T10:48:32.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.259+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f779c083770 con 0x7f77ac13ef30 2024-03-29T10:48:32.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.259+0000 7f77a1ffb640 10 client.4528.objecter ms_dispatch 0x7f77ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:32.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.259+0000 7f77a1ffb640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:32.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.259+0000 7f77a1ffb640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:32.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.259+0000 7f77a1ffb640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T10:48:32.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.259+0000 7f77b143d640 10 monclient: get_auth_request con 0x7f779006dbb0 auth_method 0 2024-03-29T10:48:32.243 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.260+0000 7f77a1ffb640 10 client.4528.objecter ms_handle_connect 0x7f779006dbb0 2024-03-29T10:48:32.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.263+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f779c04f050 con 0x7f77ac13ef30 2024-03-29T10:48:32.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.263+0000 7f77a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:32.245 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.263+0000 7f77a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:32.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.589+0000 7f77b36a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:32.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.589+0000 7f77b36a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:32.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.589+0000 7f77b36a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:32.571 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.589+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77ac1df7c0 con 0x7f77ac13ef30 2024-03-29T10:48:32.573 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:32.573 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:32.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.590+0000 7f77a1ffb640 1 -- 172.21.15.94:0/2435189268 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f779c056ba0 con 0x7f77ac13ef30 2024-03-29T10:48:32.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.590+0000 7f77a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:32.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.590+0000 7f77a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:32.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.593+0000 7f77b36a1640 10 librados: watch_flush enter 2024-03-29T10:48:32.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 10 librados: watch_flush exit 2024-03-29T10:48:32.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T10:48:32.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 10 client.4528.objecter successfully canceled tick 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 >> v1:172.21.15.94:6816/3829934808 conn(0x7f779006dbb0 legacy=0x7f7790070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 10 monclient: shutdown 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 >> v1:172.21.15.173:6789/0 conn(0x7f77ac13ef30 legacy=0x7f77ac071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 shutdown_connections 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.594+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 >> 172.21.15.94:0/2435189268 conn(0x7f77ac06b660 msgr2=0x7f77ac06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.595+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 shutdown_connections 2024-03-29T10:48:32.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.595+0000 7f77b36a1640 1 -- 172.21.15.94:0/2435189268 wait complete. 2024-03-29T10:48:32.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:32.595+0000 7f77b36a1640 1 librados: shutdown 2024-03-29T10:48:32.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:39.401 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:39.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.572+0000 7f64be0e2640 10 monclient: get_monmap_and_config 2024-03-29T10:48:39.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.572+0000 7f64be0e2640 10 monclient: build_initial_monmap 2024-03-29T10:48:39.555 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.572+0000 7f64be0e2640 10 monclient: monmap: 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:39.574512+0000 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:39.574512+0000 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:39.556 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:39.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.573+0000 7f64be0e2640 1 Processor -- start 2024-03-29T10:48:39.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.573+0000 7f64be0e2640 1 -- start start 2024-03-29T10:48:39.557 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.573+0000 7f64be0e2640 10 monclient: init 2024-03-29T10:48:39.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.574+0000 7f64be0e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:39.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.574+0000 7f64be0e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:39.558 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.574+0000 7f64be0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f64b81422b0 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.574+0000 7f64be0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f64b8142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.574+0000 7f64be0e2640 10 monclient(hunting): picked mon.noname-c con 0x7f64b80b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64be0e2640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b8142c80 con 0x7f64b8142690 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64be0e2640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b80aae40 con 0x7f64b80b17d0 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b77fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f64b81422b0 0x7f64b813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:43782/0 (socket says 172.21.15.94:43782) 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64be0e2640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b813fbd0 con 0x7f64b81422b0 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b77fe640 1 -- 172.21.15.94:0/2047594278 learned_addr learned my addr 172.21.15.94:0/2047594278 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64be0e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64be0e2640 10 monclient(hunting): authenticate will time out at 1370.844317s 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284554045 0 0) 0x7f64b80aae40 con 0x7f64b80b17d0 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b67fc640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:48:39.559 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.575+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a0003520 con 0x7f64b80b17d0 2024-03-29T10:48:39.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735901820 0 0) 0x7f64b813fbd0 con 0x7f64b81422b0 2024-03-29T10:48:39.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:48:39.560 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b80aae40 con 0x7f64b81422b0 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123107473 0 0) 0x7f64b8142c80 con 0x7f64b8142690 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b813fbd0 con 0x7f64b8142690 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516315237 0 0) 0x7f64b80aae40 con 0x7f64b81422b0 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 >> v1:172.21.15.173:6789/0 conn(0x7f64b80b17d0 legacy=0x7f64b80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 >> v1:172.21.15.94:6789/0 conn(0x7f64b8142690 legacy=0x7f64b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 monclient: found mon.noname-b 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:39.564 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8151940 con 0x7f64b81422b0 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64be0e2640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64be0e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64a8001670 con 0x7f64b81422b0 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:39.565 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: dump: 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64be0e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64a8001850 con 0x7f64b81422b0 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:39.566 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64be0e2640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.576+0000 7f64be0e2640 4 set_mon_vals no callback set 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64b67fc640 1 -- 172.21.15.94:0/2047594278 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64a8001be0 con 0x7f64b81422b0 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64b67fc640 10 monclient: dump: 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:39.567 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64be0e2640 10 monclient: shutdown 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64be0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64be0e2640 1 -- 172.21.15.94:0/2047594278 >> v1:172.21.15.94:6790/0 conn(0x7f64b81422b0 legacy=0x7f64b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64be0e2640 1 -- 172.21.15.94:0/2047594278 shutdown_connections 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64be0e2640 1 -- 172.21.15.94:0/2047594278 >> 172.21.15.94:0/2047594278 conn(0x7f64b8066a00 msgr2=0x7f64b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64be0e2640 1 -- 172.21.15.94:0/2047594278 shutdown_connections 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.577+0000 7f64be0e2640 1 -- 172.21.15.94:0/2047594278 wait complete. 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.578+0000 7f64be0e2640 10 monclient: build_initial_monmap 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.578+0000 7f64be0e2640 10 monclient: monmap: 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:39.568 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.579+0000 7f64be0e2640 1 librados: starting msgr at 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.579+0000 7f64be0e2640 1 librados: starting objecter 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.579+0000 7f64be0e2640 1 Processor -- start 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 1 -- start start 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 1 librados: setting wanted keys 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 1 librados: calling monclient init 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 10 monclient: init 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 10 monclient(hunting): picked mon.noname-c con 0x7f64b80b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f64b81422b0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:39.569 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f64b8142690 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64b77fe640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f64b80b17d0 0x7f64b812fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:44190/0 (socket says 172.21.15.94:44190) 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64b77fe640 1 -- 172.21.15.94:0/1802848315 learned_addr learned my addr 172.21.15.94:0/1802848315 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.580+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b80ac5e0 con 0x7f64b8142690 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b80ac850 con 0x7f64b81422b0 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 client.?.objecter ms_handle_connect 0x7f64b80b17d0 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b80acb30 con 0x7f64b80b17d0 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f64be0e2640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 client.?.objecter ms_handle_connect 0x7f64b81422b0 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:39.570 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f64be0e2640 10 monclient(hunting): authenticate will time out at 1370.850345s 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889148601 0 0) 0x7f64b80ac850 con 0x7f64b81422b0 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4004eb0 con 0x7f64b81422b0 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103615715 0 0) 0x7f64b80acb30 con 0x7f64b80b17d0 2024-03-29T10:48:39.573 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.581+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b80ac850 con 0x7f64b80b17d0 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 client.?.objecter ms_handle_connect 0x7f64b8142690 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2409774856 0 0) 0x7f64a4004eb0 con 0x7f64b81422b0 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b80acb30 con 0x7f64b81422b0 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64ac001890 con 0x7f64b81422b0 2024-03-29T10:48:39.574 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:39.575 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1287514950 0 0) 0x7f64b80ac850 con 0x7f64b80b17d0 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a4004eb0 con 0x7f64b80b17d0 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64a8001b60 con 0x7f64b80b17d0 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): dump: 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.576 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561749511 0 0) 0x7f64b80ac5e0 con 0x7f64b8142690 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b80ac850 con 0x7f64b8142690 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 763075966 0 0) 0x7f64b80acb30 con 0x7f64b81422b0 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 >> v1:172.21.15.94:6790/0 conn(0x7f64b80b17d0 legacy=0x7f64b812fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 >> v1:172.21.15.94:6789/0 conn(0x7f64b8142690 legacy=0x7f64b81e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:39.577 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 monclient: found mon.b 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8132250 con 0x7f64b81422b0 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient: _finish_auth 0 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f649ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 5 monclient: authenticate success, global_id 4534 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 10 monclient: _renew_subs 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b8132a10 con 0x7f64b81422b0 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 10 monclient: _renew_subs 2024-03-29T10:48:39.578 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.582+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b80ab7b0 con 0x7f64b81422b0 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.583+0000 7f64be0e2640 1 librados: init done 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.584+0000 7f64be0e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.584+0000 7f64be0e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.584+0000 7f64be0e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.584+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b81320c0 con 0x7f64b81422b0 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.585+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64ac0035f0 con 0x7f64b81422b0 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.585+0000 7f649ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.585+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f64ac003900 con 0x7f64b81422b0 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.585+0000 7f649ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.585+0000 7f649ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.585+0000 7f649ffff640 10 monclient: dump: 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:39.579 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.585+0000 7f64b57fa640 4 set_mon_vals no callback set 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.588+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f64ac0198d0 con 0x7f64b81422b0 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.588+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f64ac083780 con 0x7f64b81422b0 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.588+0000 7f649ffff640 10 client.4534.objecter ms_dispatch 0x7f64b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.588+0000 7f649ffff640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.588+0000 7f649ffff640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.588+0000 7f64b77fe640 10 monclient: get_auth_request con 0x7f64a406df90 auth_method 0 2024-03-29T10:48:39.580 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.588+0000 7f649ffff640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T10:48:39.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.589+0000 7f649ffff640 10 client.4534.objecter ms_handle_connect 0x7f64a406df90 2024-03-29T10:48:39.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.593+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64ac04f060 con 0x7f64b81422b0 2024-03-29T10:48:39.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.593+0000 7f649ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:39.581 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.593+0000 7f649ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:39.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.995+0000 7f64be0e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:39.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.995+0000 7f64be0e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:39.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.995+0000 7f64be0e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:48:39.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.995+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64b8075f10 con 0x7f64b81422b0 2024-03-29T10:48:39.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.996+0000 7f649ffff640 1 -- 172.21.15.94:0/1802848315 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f64ac050f90 con 0x7f64b81422b0 2024-03-29T10:48:39.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.996+0000 7f649ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:39.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.996+0000 7f649ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:39.979 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:39.979 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 10 librados: watch_flush enter 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 10 librados: watch_flush exit 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 10 client.4534.objecter successfully canceled tick 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 >> v1:172.21.15.94:6816/3829934808 conn(0x7f64a406df90 legacy=0x7f64a4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 10 monclient: shutdown 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:39.999+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 >> v1:172.21.15.173:6789/0 conn(0x7f64b81422b0 legacy=0x7f64b8130590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:40.000+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 shutdown_connections 2024-03-29T10:48:39.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:40.000+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 >> 172.21.15.94:0/1802848315 conn(0x7f64b8066a00 msgr2=0x7f64b813c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:39.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:40.000+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 shutdown_connections 2024-03-29T10:48:39.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:40.000+0000 7f64be0e2640 1 -- 172.21.15.94:0/1802848315 wait complete. 2024-03-29T10:48:39.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:40.000+0000 7f64be0e2640 1 librados: shutdown 2024-03-29T10:48:39.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:46.805 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.971+0000 7f0afb36d640 10 monclient: get_monmap_and_config 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.971+0000 7f0afb36d640 10 monclient: build_initial_monmap 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.971+0000 7f0afb36d640 10 monclient: monmap: 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:46.972822+0000 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:46.972822+0000 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:46.957 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:46.958 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:46.958 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:46.958 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:46.958 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.975+0000 7f0afb36d640 1 Processor -- start 2024-03-29T10:48:46.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.975+0000 7f0afb36d640 1 -- start start 2024-03-29T10:48:46.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.975+0000 7f0afb36d640 10 monclient: init 2024-03-29T10:48:46.959 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 10 monclient(hunting): picked mon.noname-b con 0x7f0af40a9ac0 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 10 monclient(hunting): picked mon.noname-a con 0x7f0af413ee90 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 10 monclient(hunting): picked mon.noname-c con 0x7f0af40af900 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af40afd10 con 0x7f0af413ee90 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af40af6f0 con 0x7f0af40af900 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af40aac80 con 0x7f0af40a9ac0 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:46.960 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.976+0000 7f0afb36d640 10 monclient(hunting): authenticate will time out at 1378.244724s 2024-03-29T10:48:46.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.981+0000 7f0af990a640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0af40af900 0x7f0af4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:42281/0 (socket says 172.21.15.94:42281) 2024-03-29T10:48:46.964 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.981+0000 7f0af990a640 1 -- 172.21.15.94:0/2160199410 learned_addr learned my addr 172.21.15.94:0/2160199410 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:46.968 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098659075 0 0) 0x7f0af40af6f0 con 0x7f0af40af900 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): my global_id is 4537 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae0003520 con 0x7f0af40af900 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058043717 0 0) 0x7f0af40afd10 con 0x7f0af413ee90 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af40af6f0 con 0x7f0af413ee90 2024-03-29T10:48:46.969 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726461279 0 0) 0x7f0af40af6f0 con 0x7f0af413ee90 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.983+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 >> v1:172.21.15.94:6790/0 conn(0x7f0af40a9ac0 legacy=0x7f0af40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 >> v1:172.21.15.173:6789/0 conn(0x7f0af40af900 legacy=0x7f0af4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: _finish_hunting 0 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 1 monclient: found mon.noname-a 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af40aaf60 con 0x7f0af413ee90 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: _finish_auth 0 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: _check_auth_tickets 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0aec001920 con 0x7f0af413ee90 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:46.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: dump: 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0aec001a80 con 0x7f0af413ee90 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 1 -- 172.21.15.94:0/2160199410 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0aec001e10 con 0x7f0af413ee90 2024-03-29T10:48:46.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0aebfff640 10 monclient: dump: 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0afb36d640 5 monclient: authenticate success, global_id 4572 2024-03-29T10:48:46.972 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0afb36d640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:46.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.984+0000 7f0afb36d640 4 set_mon_vals no callback set 2024-03-29T10:48:46.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.985+0000 7f0afb36d640 10 monclient: shutdown 2024-03-29T10:48:46.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.985+0000 7f0afb36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:46.973 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.985+0000 7f0afb36d640 1 -- 172.21.15.94:0/2160199410 >> v1:172.21.15.94:6789/0 conn(0x7f0af413ee90 legacy=0x7f0af413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.988+0000 7f0afb36d640 1 -- 172.21.15.94:0/2160199410 shutdown_connections 2024-03-29T10:48:46.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.988+0000 7f0afb36d640 1 -- 172.21.15.94:0/2160199410 >> 172.21.15.94:0/2160199410 conn(0x7f0af40b35e0 msgr2=0x7f0af40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:46.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.989+0000 7f0afb36d640 1 -- 172.21.15.94:0/2160199410 shutdown_connections 2024-03-29T10:48:46.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.989+0000 7f0afb36d640 1 -- 172.21.15.94:0/2160199410 wait complete. 2024-03-29T10:48:46.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.990+0000 7f0afb36d640 10 monclient: build_initial_monmap 2024-03-29T10:48:46.975 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.990+0000 7f0afb36d640 10 monclient: monmap: 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 1 librados: starting msgr at 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 1 librados: starting objecter 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 1 Processor -- start 2024-03-29T10:48:46.976 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 1 -- start start 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 1 librados: setting wanted keys 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 1 librados: calling monclient init 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 10 monclient: init 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.991+0000 7f0afb36d640 10 monclient(hunting): picked mon.noname-c con 0x7f0af40a9ac0 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.992+0000 7f0afb36d640 10 monclient(hunting): picked mon.noname-a con 0x7f0af40af900 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.992+0000 7f0afb36d640 10 monclient(hunting): picked mon.noname-b con 0x7f0af413ee90 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.992+0000 7f0afb36d640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af41c43d0 con 0x7f0af40af900 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.992+0000 7f0afb36d640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af41c5240 con 0x7f0af413ee90 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.992+0000 7f0afb36d640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af41c53d0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.992+0000 7f0afb36d640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:46.977 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.992+0000 7f0afb36d640 10 monclient(hunting): authenticate will time out at 1378.260164s 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.995+0000 7f0af9109640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f0af40a9ac0 0x7f0af4137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36255/0 (socket says 172.21.15.94:36255) 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.995+0000 7f0af9109640 1 -- 172.21.15.94:0/2849273070 learned_addr learned my addr 172.21.15.94:0/2849273070 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.995+0000 7f0ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f0af40a9ac0 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606992377 0 0) 0x7f0af41c53d0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad8004eb0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.982 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4037396957 0 0) 0x7f0ad8004eb0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0af41c53d0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0ae4001a30 con 0x7f0af40a9ac0 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): dump: 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:46.983 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.996+0000 7f0ae9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.997+0000 7f0ae9ffb640 10 client.?.objecter ms_handle_connect 0x7f0af40af900 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.997+0000 7f0ae9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.997+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 911507781 0 0) 0x7f0af41c53d0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.997+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 >> v1:172.21.15.173:6789/0 conn(0x7f0af413ee90 legacy=0x7f0af41de520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.997+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 >> v1:172.21.15.94:6789/0 conn(0x7f0af40af900 legacy=0x7f0af41d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 1 monclient: found mon.c 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af40722c0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:46.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0ae9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0afb36d640 5 monclient: authenticate success, global_id 4607 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0afb36d640 10 monclient: _renew_subs 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0afb36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.998+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af4072510 con 0x7f0af40a9ac0 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0afb36d640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ae4001d30 con 0x7f0af40a9ac0 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0ae9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0ae40038c0 con 0x7f0af40a9ac0 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0ae9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0ae9ffb640 10 monclient: dump: 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:46.985 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0aeaffd640 4 set_mon_vals no callback set 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0afb36d640 10 monclient: _renew_subs 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0afb36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:46.999+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af4138150 con 0x7f0af40a9ac0 2024-03-29T10:48:46.986 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.000+0000 7f0afb36d640 1 librados: init done 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.001+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f0ae4003a20 con 0x7f0af40a9ac0 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.002+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f0ae4084a50 con 0x7f0af40a9ac0 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.002+0000 7f0ae9ffb640 10 client.4607.objecter ms_dispatch 0x7f0af4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.002+0000 7f0ae9ffb640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.002+0000 7f0ae9ffb640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.002+0000 7f0ae9ffb640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.003+0000 7f0af8908640 10 monclient: get_auth_request con 0x7f0ad806dd20 auth_method 0 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.003+0000 7f0ae9ffb640 10 client.4607.objecter ms_handle_connect 0x7f0ad806dd20 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.004+0000 7f0afb36d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.004+0000 7f0afb36d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:46.988 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.004+0000 7f0afb36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:46.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.004+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af4075f10 con 0x7f0af40a9ac0 2024-03-29T10:48:46.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.014+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ae4050160 con 0x7f0af40a9ac0 2024-03-29T10:48:46.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.014+0000 7f0ae9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:46.997 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.014+0000 7f0ae9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:47.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.386+0000 7f0afb36d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:47.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.386+0000 7f0afb36d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:47.368 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.386+0000 7f0afb36d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:47.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.386+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0af406ab70 con 0x7f0af40a9ac0 2024-03-29T10:48:47.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.387+0000 7f0ae9ffb640 1 -- 172.21.15.94:0/2849273070 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f0ae4050160 con 0x7f0af40a9ac0 2024-03-29T10:48:47.369 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.387+0000 7f0ae9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:47.369 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:47.370 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:47.370 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.387+0000 7f0ae9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:47.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.390+0000 7f0afb36d640 10 librados: watch_flush enter 2024-03-29T10:48:47.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.390+0000 7f0afb36d640 10 librados: watch_flush exit 2024-03-29T10:48:47.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.390+0000 7f0afb36d640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T10:48:47.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.390+0000 7f0afb36d640 10 client.4607.objecter successfully canceled tick 2024-03-29T10:48:47.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.391+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 >> v1:172.21.15.94:6816/3829934808 conn(0x7f0ad806dd20 legacy=0x7f0ad80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.391+0000 7f0afb36d640 10 monclient: shutdown 2024-03-29T10:48:47.373 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.391+0000 7f0afb36d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:47.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.391+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 >> v1:172.21.15.94:6790/0 conn(0x7f0af40a9ac0 legacy=0x7f0af4137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:47.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.391+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 shutdown_connections 2024-03-29T10:48:47.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.391+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 >> 172.21.15.94:0/2849273070 conn(0x7f0af40b35e0 msgr2=0x7f0af413af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:47.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.392+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 shutdown_connections 2024-03-29T10:48:47.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.392+0000 7f0afb36d640 1 -- 172.21.15.94:0/2849273070 wait complete. 2024-03-29T10:48:47.374 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:47.393+0000 7f0afb36d640 1 librados: shutdown 2024-03-29T10:48:47.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:54.197 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:54.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.354+0000 7f011c7db640 10 monclient: get_monmap_and_config 2024-03-29T10:48:54.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.354+0000 7f011c7db640 10 monclient: build_initial_monmap 2024-03-29T10:48:54.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.354+0000 7f011c7db640 10 monclient: monmap: 2024-03-29T10:48:54.337 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:54.337 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:48:54.355493+0000 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:48:54.355493+0000 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.355+0000 7f011c7db640 1 Processor -- start 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.355+0000 7f011c7db640 1 -- start start 2024-03-29T10:48:54.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.355+0000 7f011c7db640 10 monclient: init 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 10 monclient(hunting): picked mon.noname-c con 0x7f01140ab360 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 10 monclient(hunting): picked mon.noname-a con 0x7f01140ad6c0 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 10 monclient(hunting): picked mon.noname-b con 0x7f0114153090 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140a94e0 con 0x7f01140ad6c0 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140af3a0 con 0x7f01140ab360 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140afb60 con 0x7f0114153090 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:54.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.356+0000 7f011c7db640 10 monclient(hunting): authenticate will time out at 1385.624650s 2024-03-29T10:48:54.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.358+0000 7f011ad78640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f0114153090 0x7f01141538a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41058/0 (socket says 172.21.15.94:41058) 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.358+0000 7f011ad78640 1 -- 172.21.15.94:0/1771290215 learned_addr learned my addr 172.21.15.94:0/1771290215 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.358+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019174162 0 0) 0x7f01140afb60 con 0x7f0114153090 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 10 monclient(hunting): _negotiate 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0100003520 con 0x7f0114153090 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859284606 0 0) 0x7f0100003520 con 0x7f0114153090 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 >> v1:172.21.15.173:6789/0 conn(0x7f01140ab360 legacy=0x7f011414fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:54.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.359+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 >> v1:172.21.15.94:6789/0 conn(0x7f01140ad6c0 legacy=0x7f0114152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:54.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: _finish_hunting 0 2024-03-29T10:48:54.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 1 monclient: found mon.noname-b 2024-03-29T10:48:54.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:54.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:48:54.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140aab80 con 0x7f0114153090 2024-03-29T10:48:54.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: _finish_auth 0 2024-03-29T10:48:54.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: _check_auth_tickets 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f011c7db640 5 monclient: authenticate success, global_id 4610 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f011c7db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f010c001a10 con 0x7f0114153090 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: dump: 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:54.344 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f011c7db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f010c001d20 con 0x7f0114153090 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f0119575640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f011c7db640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.360+0000 7f011c7db640 4 set_mon_vals no callback set 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f0119575640 1 -- 172.21.15.94:0/1771290215 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f010c0020b0 con 0x7f0114153090 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f0119575640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f0119575640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f0119575640 10 monclient: dump: 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:54.345 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f011c7db640 10 monclient: shutdown 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f011c7db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:54.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f011c7db640 1 -- 172.21.15.94:0/1771290215 >> v1:172.21.15.94:6790/0 conn(0x7f0114153090 legacy=0x7f01141538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f011c7db640 1 -- 172.21.15.94:0/1771290215 shutdown_connections 2024-03-29T10:48:54.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.361+0000 7f011c7db640 1 -- 172.21.15.94:0/1771290215 >> 172.21.15.94:0/1771290215 conn(0x7f01140b35e0 msgr2=0x7f01140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:54.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.362+0000 7f011c7db640 1 -- 172.21.15.94:0/1771290215 shutdown_connections 2024-03-29T10:48:54.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.362+0000 7f011c7db640 1 -- 172.21.15.94:0/1771290215 wait complete. 2024-03-29T10:48:54.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 10 monclient: build_initial_monmap 2024-03-29T10:48:54.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 10 monclient: monmap: 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 1 librados: starting msgr at 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 1 librados: starting objecter 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 1 Processor -- start 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 1 -- start start 2024-03-29T10:48:54.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 1 librados: setting wanted keys 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 1 librados: calling monclient init 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.363+0000 7f011c7db640 10 monclient: init 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 10 monclient(hunting): picked mon.noname-a con 0x7f01140ab360 addr v1:172.21.15.94:6789/0 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 10 monclient(hunting): picked mon.noname-b con 0x7f01140ad6c0 addr v1:172.21.15.173:6789/0 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 10 monclient(hunting): picked mon.noname-c con 0x7f0114153090 addr v1:172.21.15.94:6790/0 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140b1300 con 0x7f01140ab360 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114071b00 con 0x7f01140ad6c0 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01140b2e30 con 0x7f0114153090 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.364+0000 7f011c7db640 10 monclient(hunting): authenticate will time out at 1385.632453s 2024-03-29T10:48:54.350 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.365+0000 7f011ad78640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f0114153090 0x7f01141f34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41332/0 (socket says 172.21.15.94:41332) 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.365+0000 7f011ad78640 1 -- 172.21.15.94:0/3877360715 learned_addr learned my addr 172.21.15.94:0/3877360715 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.365+0000 7f010affd640 10 client.?.objecter ms_handle_connect 0x7f0114153090 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 client.?.objecter ms_handle_connect 0x7f01140ad6c0 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878074845 0 0) 0x7f01140b2e30 con 0x7f0114153090 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f8004eb0 con 0x7f0114153090 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578643829 0 0) 0x7f0114071b00 con 0x7f01140ad6c0 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): _negotiate 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:54.351 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 10 monclient(hunting): my global_id is 4540 2024-03-29T10:48:54.352 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.366+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01140b2e30 con 0x7f01140ad6c0 2024-03-29T10:48:54.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.367+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 951900248 0 0) 0x7f00f8004eb0 con 0x7f0114153090 2024-03-29T10:48:54.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.367+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0114071b00 con 0x7f0114153090 2024-03-29T10:48:54.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.367+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f010c001a50 con 0x7f0114153090 2024-03-29T10:48:54.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.367+0000 7f010affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.367+0000 7f010affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:54.353 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.367+0000 7f010affd640 10 monclient(hunting): dump: 2024-03-29T10:48:54.353 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.367+0000 7f010affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.368+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4092058409 0 0) 0x7f01140b2e30 con 0x7f01140ad6c0 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.368+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00f8004eb0 con 0x7f01140ad6c0 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.368+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f0110001830 con 0x7f01140ad6c0 2024-03-29T10:48:54.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.368+0000 7f010affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.368+0000 7f010affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.368+0000 7f010affd640 10 monclient(hunting): dump: 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3957240399 0 0) 0x7f0114071b00 con 0x7f0114153090 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 >> v1:172.21.15.173:6789/0 conn(0x7f01140ad6c0 legacy=0x7f01141efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 >> v1:172.21.15.94:6789/0 conn(0x7f01140ab360 legacy=0x7f01140b1790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 10 monclient: _finish_hunting 0 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 1 monclient: found mon.c 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140b3110 con 0x7f0114153090 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 10 monclient: _finish_auth 0 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 10 monclient: _check_auth_tickets 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f010affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 5 monclient: authenticate success, global_id 4613 2024-03-29T10:48:54.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 10 monclient: _renew_subs 2024-03-29T10:48:54.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:54.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0114072560 con 0x7f0114153090 2024-03-29T10:48:54.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:54.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 10 monclient: _renew_subs 2024-03-29T10:48:54.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:54.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f011406ecf0 con 0x7f0114153090 2024-03-29T10:48:54.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.369+0000 7f011c7db640 1 librados: init done 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.371+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f010c004e10 con 0x7f0114153090 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.371+0000 7f010affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.371+0000 7f0118d74640 4 set_mon_vals no callback set 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.371+0000 7f011c7db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.372+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f010c005220 con 0x7f0114153090 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.372+0000 7f010affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.372+0000 7f010affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.372+0000 7f010affd640 10 monclient: dump: 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.358 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.372+0000 7f011c7db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.372+0000 7f011c7db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.372+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc005080 con 0x7f0114153090 2024-03-29T10:48:54.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.373+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f010c005440 con 0x7f0114153090 2024-03-29T10:48:54.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.374+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 348853018 0 0) 0x7f010c084ee0 con 0x7f0114153090 2024-03-29T10:48:54.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.374+0000 7f010affd640 10 client.4613.objecter ms_dispatch 0x7f0114000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:54.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.374+0000 7f010affd640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:54.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.374+0000 7f010affd640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:54.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.374+0000 7f010affd640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T10:48:54.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.377+0000 7f011a577640 10 monclient: get_auth_request con 0x7f00f806de90 auth_method 0 2024-03-29T10:48:54.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.377+0000 7f010affd640 10 client.4613.objecter ms_handle_connect 0x7f00f806de90 2024-03-29T10:48:54.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.381+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f010c0507c0 con 0x7f0114153090 2024-03-29T10:48:54.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.381+0000 7f010affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:54.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.381+0000 7f010affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:54.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.705+0000 7f011c7db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:54.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.705+0000 7f011c7db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:54.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.705+0000 7f011c7db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:48:54.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.705+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00fc005330 con 0x7f0114153090 2024-03-29T10:48:54.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.707+0000 7f010affd640 1 -- 172.21.15.94:0/3877360715 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 595538824) 0x7f010c0526f0 con 0x7f0114153090 2024-03-29T10:48:54.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.707+0000 7f010affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:54.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.707+0000 7f010affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:54.692 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:48:54.692 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":10,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:38:57.932349+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:38:57.932349+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 10 librados: watch_flush enter 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 10 librados: watch_flush exit 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 10 client.4613.objecter successfully canceled tick 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 >> v1:172.21.15.94:6816/3829934808 conn(0x7f00f806de90 legacy=0x7f00f8070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 10 monclient: shutdown 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 >> v1:172.21.15.94:6790/0 conn(0x7f0114153090 legacy=0x7f01141f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 shutdown_connections 2024-03-29T10:48:54.693 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.711+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 >> 172.21.15.94:0/3877360715 conn(0x7f01140b35e0 msgr2=0x7f01140b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:54.694 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.712+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 shutdown_connections 2024-03-29T10:48:54.695 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.712+0000 7f011c7db640 1 -- 172.21.15.94:0/3877360715 wait complete. 2024-03-29T10:48:54.695 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:48:54.714+0000 7f011c7db640 1 librados: shutdown 2024-03-29T10:48:54.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:01.520 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:01.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.670+0000 7f47143cd640 10 monclient: get_monmap_and_config 2024-03-29T10:49:01.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.670+0000 7f47143cd640 10 monclient: build_initial_monmap 2024-03-29T10:49:01.654 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.670+0000 7f47143cd640 10 monclient: monmap: 2024-03-29T10:49:01.654 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:01.655 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:01.655 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:01.671739+0000 2024-03-29T10:49:01.655 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:01.671739+0000 2024-03-29T10:49:01.655 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:01.655 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:01.655 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:01.655 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:01.656 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:01.656 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:01.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.671+0000 7f47143cd640 1 Processor -- start 2024-03-29T10:49:01.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.672+0000 7f47143cd640 1 -- start start 2024-03-29T10:49:01.656 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.672+0000 7f47143cd640 10 monclient: init 2024-03-29T10:49:01.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:01.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:01.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 10 monclient(hunting): picked mon.noname-b con 0x7f470c0ae250 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:01.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 10 monclient(hunting): picked mon.noname-a con 0x7f470c13f970 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:01.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 10 monclient(hunting): picked mon.noname-c con 0x7f470c142800 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:01.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c0afe00 con 0x7f470c13f970 2024-03-29T10:49:01.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c142c10 con 0x7f470c142800 2024-03-29T10:49:01.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c0afb20 con 0x7f470c0ae250 2024-03-29T10:49:01.659 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:01.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f47143cd640 10 monclient(hunting): authenticate will time out at 1392.941292s 2024-03-29T10:49:01.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4712169640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f470c0ae250 0x7f470c13f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:41050/0 (socket says 172.21.15.94:41050) 2024-03-29T10:49:01.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4712169640 1 -- 172.21.15.94:0/4175307949 learned_addr learned my addr 172.21.15.94:0/4175307949 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:01.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643831970 0 0) 0x7f470c0afb20 con 0x7f470c0ae250 2024-03-29T10:49:01.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4711167640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.660 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4711167640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4711167640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4711167640 10 monclient(hunting): my global_id is 4616 2024-03-29T10:49:01.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.673+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f8003520 con 0x7f470c0ae250 2024-03-29T10:49:01.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.674+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715683310 0 0) 0x7f46f8003520 con 0x7f470c0ae250 2024-03-29T10:49:01.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.674+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 >> v1:172.21.15.173:6789/0 conn(0x7f470c142800 legacy=0x7f470c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:01.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.674+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 >> v1:172.21.15.94:6789/0 conn(0x7f470c13f970 legacy=0x7f470c1420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:01.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: _finish_hunting 0 2024-03-29T10:49:01.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 1 monclient: found mon.noname-b 2024-03-29T10:49:01.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:01.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:49:01.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c0aab80 con 0x7f470c0ae250 2024-03-29T10:49:01.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: _finish_auth 0 2024-03-29T10:49:01.672 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: _check_auth_tickets 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4708001690 con 0x7f470c0ae250 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: dump: 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:01.673 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f47143cd640 5 monclient: authenticate success, global_id 4616 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f47143cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 1 -- 172.21.15.94:0/4175307949 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4708001860 con 0x7f470c0ae250 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f4711167640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f47143cd640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.677+0000 7f47143cd640 4 set_mon_vals no callback set 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.678+0000 7f47143cd640 10 monclient: shutdown 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.678+0000 7f47143cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.678+0000 7f47143cd640 1 -- 172.21.15.94:0/4175307949 >> v1:172.21.15.94:6790/0 conn(0x7f470c0ae250 legacy=0x7f470c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.674 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.678+0000 7f47143cd640 1 -- 172.21.15.94:0/4175307949 shutdown_connections 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.679+0000 7f47143cd640 1 -- 172.21.15.94:0/4175307949 >> 172.21.15.94:0/4175307949 conn(0x7f470c0b35e0 msgr2=0x7f470c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.679+0000 7f47143cd640 1 -- 172.21.15.94:0/4175307949 shutdown_connections 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.679+0000 7f47143cd640 1 -- 172.21.15.94:0/4175307949 wait complete. 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 10 monclient: build_initial_monmap 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 10 monclient: monmap: 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:01.675 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 1 librados: starting msgr at 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 1 librados: starting objecter 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 1 Processor -- start 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 1 -- start start 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 1 librados: setting wanted keys 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 1 librados: calling monclient init 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.680+0000 7f47143cd640 10 monclient: init 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:01.676 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 10 monclient(hunting): picked mon.noname-b con 0x7f470c0ae250 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 10 monclient(hunting): picked mon.noname-c con 0x7f470c13f970 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f4711968640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f470c13f970 0x7f470c144d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:42334/0 (socket says 172.21.15.94:42334) 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f4711968640 1 -- 172.21.15.94:0/3819065822 learned_addr learned my addr 172.21.15.94:0/3819065822 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 10 monclient(hunting): picked mon.noname-a con 0x7f470c142800 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c13c550 con 0x7f470c142800 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c1db100 con 0x7f470c0ae250 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470c1dadd0 con 0x7f470c13f970 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f47143cd640 10 monclient(hunting): authenticate will time out at 1392.949992s 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f4702ffd640 10 client.?.objecter ms_handle_connect 0x7f470c13f970 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.681+0000 7f4702ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:01.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.682+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366423406 0 0) 0x7f470c1dadd0 con 0x7f470c13f970 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.682+0000 7f4702ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.682+0000 7f4702ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.682+0000 7f4702ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.682+0000 7f4702ffd640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.682+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f0004eb0 con 0x7f470c13f970 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.682+0000 7f4702ffd640 10 client.?.objecter ms_handle_connect 0x7f470c0ae250 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128220937 0 0) 0x7f470c1db100 con 0x7f470c0ae250 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:01.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c1dadd0 con 0x7f470c0ae250 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 449002819 0 0) 0x7f46f0004eb0 con 0x7f470c13f970 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f470c1db100 con 0x7f470c13f970 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f46fc001960 con 0x7f470c13f970 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:01.679 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.683+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970556256 0 0) 0x7f470c1dadd0 con 0x7f470c0ae250 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.684+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f0004eb0 con 0x7f470c0ae250 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.684+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3185092719 0 0) 0x7f470c1db100 con 0x7f470c13f970 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.684+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 >> v1:172.21.15.173:6789/0 conn(0x7f470c0ae250 legacy=0x7f470c1da6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.684+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 >> v1:172.21.15.94:6789/0 conn(0x7f470c142800 legacy=0x7f470c13a520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:01.680 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 1 monclient: found mon.c 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c145b10 con 0x7f470c13f970 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 10 monclient: _renew_subs 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c145d00 con 0x7f470c13f970 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:01.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 10 monclient: _renew_subs 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f470c145660 con 0x7f470c13f970 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f47143cd640 1 librados: init done 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46fc0035f0 con 0x7f470c13f970 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4710966640 4 set_mon_vals no callback set 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f46fc003940 con 0x7f470c13f970 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.685+0000 7f4702ffd640 10 monclient: dump: 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:01.682 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:01.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.687+0000 7f47143cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.687+0000 7f47143cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.687+0000 7f47143cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.687+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470c1daf60 con 0x7f470c13f970 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f46fc003aa0 con 0x7f470c13f970 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7f46fc0849c0 con 0x7f470c13f970 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4712169640 10 monclient: get_auth_request con 0x7f46f006dd20 auth_method 0 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4702ffd640 10 client.4619.objecter ms_dispatch 0x7f470c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4702ffd640 3 client.4619.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4702ffd640 3 client.4619.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4702ffd640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T10:49:01.684 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.688+0000 7f4702ffd640 10 client.4619.objecter ms_handle_connect 0x7f46f006dd20 2024-03-29T10:49:01.685 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.696+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f470c1daf60 con 0x7f470c13f970 2024-03-29T10:49:01.685 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.696+0000 7f4702ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:01.685 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:01.696+0000 7f4702ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:01.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.007+0000 7f47143cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.007+0000 7f47143cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.007+0000 7f47143cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:01.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.007+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f470c1daf60 con 0x7f470c13f970 2024-03-29T10:49:01.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.008+0000 7f4702ffd640 1 -- 172.21.15.94:0/3819065822 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7f470c1daf60 con 0x7f470c13f970 2024-03-29T10:49:01.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.008+0000 7f4702ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:01.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.008+0000 7f4702ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:01.991 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:01.991 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.011+0000 7f47143cd640 10 librados: watch_flush enter 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.012+0000 7f47143cd640 10 librados: watch_flush exit 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.012+0000 7f47143cd640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.012+0000 7f47143cd640 10 client.4619.objecter successfully canceled tick 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.012+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 >> v1:172.21.15.94:6816/3829934808 conn(0x7f46f006dd20 legacy=0x7f46f00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.012+0000 7f47143cd640 10 monclient: shutdown 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.012+0000 7f47143cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:01.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.012+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 >> v1:172.21.15.94:6790/0 conn(0x7f470c13f970 legacy=0x7f470c144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:01.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.013+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 shutdown_connections 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.013+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 >> 172.21.15.94:0/3819065822 conn(0x7f470c0b35e0 msgr2=0x7f470c13e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.013+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 shutdown_connections 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.013+0000 7f47143cd640 1 -- 172.21.15.94:0/3819065822 wait complete. 2024-03-29T10:49:01.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:02.014+0000 7f47143cd640 1 librados: shutdown 2024-03-29T10:49:02.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:08.823 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:08.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.984+0000 7fbc5862e640 10 monclient: get_monmap_and_config 2024-03-29T10:49:08.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.985+0000 7fbc5862e640 10 monclient: build_initial_monmap 2024-03-29T10:49:08.967 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.985+0000 7fbc5862e640 10 monclient: monmap: 2024-03-29T10:49:08.967 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:08.967 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:08.967 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:08.985887+0000 2024-03-29T10:49:08.968 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:08.985887+0000 2024-03-29T10:49:08.968 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:08.968 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:08.968 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:08.968 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:08.968 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:08.968 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:08.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.987+0000 7fbc5862e640 1 Processor -- start 2024-03-29T10:49:08.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.988+0000 7fbc5862e640 1 -- start start 2024-03-29T10:49:08.970 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.988+0000 7fbc5862e640 10 monclient: init 2024-03-29T10:49:08.971 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc500a9930 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc50149dc0 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc5014ff40 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc500aae60 con 0x7fbc500a9930 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc500b4310 con 0x7fbc5014ff40 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc500afdd0 con 0x7fbc50149dc0 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.989+0000 7fbc5862e640 10 monclient(hunting): authenticate will time out at 1400.257287s 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc55bc9640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fbc50149dc0 0x7fbc5014a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:36844/0 (socket says 172.21.15.94:36844) 2024-03-29T10:49:08.978 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc55bc9640 1 -- 172.21.15.94:0/2385282846 learned_addr learned my addr 172.21.15.94:0/2385282846 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631884417 0 0) 0x7fbc500afdd0 con 0x7fbc50149dc0 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc553c8640 10 monclient(hunting): _negotiate 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc553c8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc553c8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc553c8640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.992+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc3c003520 con 0x7fbc50149dc0 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746206743 0 0) 0x7fbc3c003520 con 0x7fbc50149dc0 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 >> v1:172.21.15.173:6789/0 conn(0x7fbc5014ff40 legacy=0x7fbc50152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 >> v1:172.21.15.94:6789/0 conn(0x7fbc500a9930 legacy=0x7fbc50143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: _finish_hunting 0 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 1 monclient: found mon.noname-b 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:49:08.979 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5014a950 con 0x7fbc50149dc0 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: _finish_auth 0 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: _check_auth_tickets 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc5862e640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc5862e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fbc48001a10 con 0x7fbc50149dc0 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: dump: 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:08.980 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc5862e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 1 -- 172.21.15.94:0/2385282846 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc48001ca0 con 0x7fbc50149dc0 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc553c8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc5862e640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.993+0000 7fbc5862e640 4 set_mon_vals no callback set 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.994+0000 7fbc5862e640 10 monclient: shutdown 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.994+0000 7fbc5862e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.994+0000 7fbc5862e640 1 -- 172.21.15.94:0/2385282846 >> v1:172.21.15.94:6790/0 conn(0x7fbc50149dc0 legacy=0x7fbc5014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.995+0000 7fbc5862e640 1 -- 172.21.15.94:0/2385282846 shutdown_connections 2024-03-29T10:49:08.981 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.995+0000 7fbc5862e640 1 -- 172.21.15.94:0/2385282846 >> 172.21.15.94:0/2385282846 conn(0x7fbc500b35e0 msgr2=0x7fbc500b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:08.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.997+0000 7fbc5862e640 1 -- 172.21.15.94:0/2385282846 shutdown_connections 2024-03-29T10:49:08.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.997+0000 7fbc5862e640 1 -- 172.21.15.94:0/2385282846 wait complete. 2024-03-29T10:49:08.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.998+0000 7fbc5862e640 10 monclient: build_initial_monmap 2024-03-29T10:49:08.983 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.998+0000 7fbc5862e640 10 monclient: monmap: 2024-03-29T10:49:08.983 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.998+0000 7fbc5862e640 1 librados: starting msgr at 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.998+0000 7fbc5862e640 1 librados: starting objecter 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:08.998+0000 7fbc5862e640 1 Processor -- start 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.000+0000 7fbc5862e640 1 -- start start 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.000+0000 7fbc5862e640 1 librados: setting wanted keys 2024-03-29T10:49:08.984 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.000+0000 7fbc5862e640 1 librados: calling monclient init 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.000+0000 7fbc5862e640 10 monclient: init 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 10 monclient(hunting): picked mon.noname-b con 0x7fbc500a9930 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 10 monclient(hunting): picked mon.noname-a con 0x7fbc50149dc0 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 10 monclient(hunting): picked mon.noname-c con 0x7fbc5014ff40 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc500ab990 con 0x7fbc50149dc0 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc500abb20 con 0x7fbc500a9930 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc501389b0 con 0x7fbc5014ff40 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:08.985 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.001+0000 7fbc5862e640 10 monclient(hunting): authenticate will time out at 1400.269734s 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc56bcb640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7fbc5014ff40 0x7fbc501382a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:38196/0 (socket says 172.21.15.94:38196) 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc56bcb640 1 -- 172.21.15.94:0/3394497368 learned_addr learned my addr 172.21.15.94:0/3394497368 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 10 client.?.objecter ms_handle_connect 0x7fbc5014ff40 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057903194 0 0) 0x7fbc501389b0 con 0x7fbc5014ff40 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:08.989 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.003+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc34004eb0 con 0x7fbc5014ff40 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.004+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3631073265 0 0) 0x7fbc34004eb0 con 0x7fbc5014ff40 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.004+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc501389b0 con 0x7fbc5014ff40 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fbc40001760 con 0x7fbc5014ff40 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:08.990 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 60021100 0 0) 0x7fbc501389b0 con 0x7fbc5014ff40 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 >> v1:172.21.15.173:6789/0 conn(0x7fbc500a9930 legacy=0x7fbc501414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 >> v1:172.21.15.94:6789/0 conn(0x7fbc50149dc0 legacy=0x7fbc50149170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 1 monclient: found mon.c 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc50138b40 con 0x7fbc5014ff40 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.005+0000 7fbc46ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:08.991 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 10 monclient: _renew_subs 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc500ac5a0 con 0x7fbc5014ff40 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 10 monclient: _renew_subs 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc500acad0 con 0x7fbc5014ff40 2024-03-29T10:49:08.992 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.006+0000 7fbc5862e640 1 librados: init done 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.009+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc40001d30 con 0x7fbc5014ff40 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.009+0000 7fbc46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.009+0000 7fbc54bc7640 4 set_mon_vals no callback set 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.010+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7fbc400049b0 con 0x7fbc5014ff40 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.010+0000 7fbc46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.010+0000 7fbc46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.010+0000 7fbc46ffd640 10 monclient: dump: 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:08.994 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.010+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7fbc4001aa00 con 0x7fbc5014ff40 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc5862e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc5862e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc5862e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc38005080 con 0x7fbc5014ff40 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7fbc40084aa0 con 0x7fbc5014ff40 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc46ffd640 10 client.4625.objecter ms_dispatch 0x7fbc50000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:08.995 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc46ffd640 3 client.4625.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:08.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc46ffd640 3 client.4625.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:08.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.011+0000 7fbc46ffd640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T10:49:08.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.012+0000 7fbc563ca640 10 monclient: get_auth_request con 0x7fbc3406dc60 auth_method 0 2024-03-29T10:49:08.996 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.012+0000 7fbc46ffd640 10 client.4625.objecter ms_handle_connect 0x7fbc3406dc60 2024-03-29T10:49:09.005 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.023+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc40050400 con 0x7fbc5014ff40 2024-03-29T10:49:09.005 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.023+0000 7fbc46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:09.005 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.023+0000 7fbc46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:09.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.350+0000 7fbc5862e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:09.332 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.350+0000 7fbc5862e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:09.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.350+0000 7fbc5862e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:09.333 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.350+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc38005330 con 0x7fbc5014ff40 2024-03-29T10:49:09.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.354+0000 7fbc46ffd640 1 -- 172.21.15.94:0/3394497368 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7fbc40052330 con 0x7fbc5014ff40 2024-03-29T10:49:09.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.354+0000 7fbc46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:09.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.354+0000 7fbc46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:09.339 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:09.339 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 10 librados: watch_flush enter 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 10 librados: watch_flush exit 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 10 client.4625.objecter successfully canceled tick 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 >> v1:172.21.15.94:6816/3829934808 conn(0x7fbc3406dc60 legacy=0x7fbc34070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 10 monclient: shutdown 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:09.340 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 >> v1:172.21.15.94:6790/0 conn(0x7fbc5014ff40 legacy=0x7fbc501382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:09.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 shutdown_connections 2024-03-29T10:49:09.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.358+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 >> 172.21.15.94:0/3394497368 conn(0x7fbc500b35e0 msgr2=0x7fbc500b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:09.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.361+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 shutdown_connections 2024-03-29T10:49:09.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.361+0000 7fbc5862e640 1 -- 172.21.15.94:0/3394497368 wait complete. 2024-03-29T10:49:09.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:09.361+0000 7fbc5862e640 1 librados: shutdown 2024-03-29T10:49:09.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:16.166 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.354+0000 7f74b898e640 10 monclient: get_monmap_and_config 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.354+0000 7f74b898e640 10 monclient: build_initial_monmap 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.354+0000 7f74b898e640 10 monclient: monmap: 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:16.355252+0000 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:16.355252+0000 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:16.336 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:16.337 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:16.337 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:16.337 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:16.337 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:16.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.355+0000 7f74b898e640 1 Processor -- start 2024-03-29T10:49:16.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.355+0000 7f74b898e640 1 -- start start 2024-03-29T10:49:16.337 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.355+0000 7f74b898e640 10 monclient: init 2024-03-29T10:49:16.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:16.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:16.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 10 monclient(hunting): picked mon.noname-a con 0x7f74b41422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:16.338 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 10 monclient(hunting): picked mon.noname-c con 0x7f74b4142690 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 10 monclient(hunting): picked mon.noname-b con 0x7f74b40b17d0 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b4142c80 con 0x7f74b41422b0 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40aae40 con 0x7f74b4142690 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b413fbd0 con 0x7f74b40b17d0 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b898e640 10 monclient(hunting): authenticate will time out at 1407.624725s 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b2d9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f74b40b17d0 0x7f74b40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:48944/0 (socket says 172.21.15.94:48944) 2024-03-29T10:49:16.339 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.356+0000 7f74b2d9d640 1 -- 172.21.15.94:0/2720801444 learned_addr learned my addr 172.21.15.94:0/2720801444 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595114661 0 0) 0x7f74b413fbd0 con 0x7f74b40b17d0 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f749c003520 con 0x7f74b40b17d0 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445998514 0 0) 0x7f749c003520 con 0x7f74b40b17d0 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.357+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 >> v1:172.21.15.173:6789/0 conn(0x7f74b4142690 legacy=0x7f74b40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 >> v1:172.21.15.94:6789/0 conn(0x7f74b41422b0 legacy=0x7f74b413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 1 monclient: found mon.noname-b 2024-03-29T10:49:16.341 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4151940 con 0x7f74b40b17d0 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 10 monclient: _finish_auth 0 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74ac001a10 con 0x7f74b40b17d0 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b159a640 10 monclient: dump: 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.342 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b898e640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.358+0000 7f74b898e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74ac001c70 con 0x7f74b40b17d0 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b159a640 1 -- 172.21.15.94:0/2720801444 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74ac002040 con 0x7f74b40b17d0 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b898e640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b898e640 4 set_mon_vals no callback set 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.343 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:16.344 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b159a640 10 monclient: dump: 2024-03-29T10:49:16.344 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:16.344 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:16.344 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.344 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.344 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.344 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:16.345 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:16.345 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:16.345 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:16.345 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b898e640 10 monclient: shutdown 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b898e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.359+0000 7f74b898e640 1 -- 172.21.15.94:0/2720801444 >> v1:172.21.15.94:6790/0 conn(0x7f74b40b17d0 legacy=0x7f74b40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.360+0000 7f74b898e640 1 -- 172.21.15.94:0/2720801444 shutdown_connections 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.360+0000 7f74b898e640 1 -- 172.21.15.94:0/2720801444 >> 172.21.15.94:0/2720801444 conn(0x7f74b4066a00 msgr2=0x7f74b4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.361+0000 7f74b898e640 1 -- 172.21.15.94:0/2720801444 shutdown_connections 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.362+0000 7f74b898e640 1 -- 172.21.15.94:0/2720801444 wait complete. 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.363+0000 7f74b898e640 10 monclient: build_initial_monmap 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.363+0000 7f74b898e640 10 monclient: monmap: 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:16.346 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.364+0000 7f74b898e640 1 librados: starting msgr at 2024-03-29T10:49:16.347 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.364+0000 7f74b898e640 1 librados: starting objecter 2024-03-29T10:49:16.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.364+0000 7f74b898e640 1 Processor -- start 2024-03-29T10:49:16.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.366+0000 7f74b898e640 1 -- start start 2024-03-29T10:49:16.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.366+0000 7f74b898e640 1 librados: setting wanted keys 2024-03-29T10:49:16.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.366+0000 7f74b898e640 1 librados: calling monclient init 2024-03-29T10:49:16.348 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.366+0000 7f74b898e640 10 monclient: init 2024-03-29T10:49:16.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.367+0000 7f74b898e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:16.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.367+0000 7f74b898e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:16.349 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 10 monclient(hunting): picked mon.noname-b con 0x7f74b40b17d0 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:16.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b259c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f74b40b17d0 0x7f74b40b5780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:37716/0 (socket says 172.21.15.94:37716) 2024-03-29T10:49:16.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b259c640 1 -- 172.21.15.94:0/13918342 learned_addr learned my addr 172.21.15.94:0/13918342 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:16.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 10 monclient(hunting): picked mon.noname-a con 0x7f74b41422b0 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:16.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74a2ffd640 10 client.?.objecter ms_handle_connect 0x7f74b40b17d0 2024-03-29T10:49:16.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:16.354 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 10 monclient(hunting): picked mon.noname-c con 0x7f74b4142690 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40b5e90 con 0x7f74b41422b0 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b412d9b0 con 0x7f74b40b17d0 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40ac9a0 con 0x7f74b4142690 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.368+0000 7f74b898e640 10 monclient(hunting): authenticate will time out at 1407.637013s 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 client.?.objecter ms_handle_connect 0x7f74b41422b0 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946735340 0 0) 0x7f74b412d9b0 con 0x7f74b40b17d0 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 monclient(hunting): my global_id is 4546 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7494004eb0 con 0x7f74b40b17d0 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 client.?.objecter ms_handle_connect 0x7f74b4142690 2024-03-29T10:49:16.355 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.369+0000 7f74a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976291877 0 0) 0x7f74b40b5e90 con 0x7f74b41422b0 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): my global_id is 4578 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b412d9b0 con 0x7f74b41422b0 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227870784 0 0) 0x7f74b40ac9a0 con 0x7f74b4142690 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b40b5e90 con 0x7f74b4142690 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226009153 0 0) 0x7f7494004eb0 con 0x7f74b40b17d0 2024-03-29T10:49:16.356 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b40ac9a0 con 0x7f74b40b17d0 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74a4001960 con 0x7f74b40b17d0 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:16.357 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.370+0000 7f74a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2918975761 0 0) 0x7f74b40b5e90 con 0x7f74b4142690 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7494004eb0 con 0x7f74b4142690 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74ac001de0 con 0x7f74b4142690 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:16.358 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 815165485 0 0) 0x7f74b40ac9a0 con 0x7f74b40b17d0 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 >> v1:172.21.15.94:6790/0 conn(0x7f74b4142690 legacy=0x7f74b412d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 >> v1:172.21.15.94:6789/0 conn(0x7f74b41422b0 legacy=0x7f74b412cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 monclient: found mon.b 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b40acc80 con 0x7f74b40b17d0 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74b898e640 5 monclient: authenticate success, global_id 4546 2024-03-29T10:49:16.359 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74b898e640 10 monclient: _renew_subs 2024-03-29T10:49:16.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74b898e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:16.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b40acf60 con 0x7f74b40b17d0 2024-03-29T10:49:16.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.371+0000 7f74b898e640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:16.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.372+0000 7f74b898e640 10 monclient: _renew_subs 2024-03-29T10:49:16.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.372+0000 7f74b898e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:16.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.372+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b40b6da0 con 0x7f74b40b17d0 2024-03-29T10:49:16.360 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.372+0000 7f74b898e640 1 librados: init done 2024-03-29T10:49:16.361 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.374+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74a4001ac0 con 0x7f74b40b17d0 2024-03-29T10:49:16.361 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.374+0000 7f74a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:16.361 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.374+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f74a40038a0 con 0x7f74b40b17d0 2024-03-29T10:49:16.361 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.374+0000 7f74a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:16.361 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.374+0000 7f74a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.374+0000 7f74a2ffd640 10 monclient: dump: 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.374+0000 7f74b0d99640 4 set_mon_vals no callback set 2024-03-29T10:49:16.362 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.375+0000 7f74b898e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.375+0000 7f74b898e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.375+0000 7f74b898e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.375+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7498005080 con 0x7f74b40b17d0 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.376+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f74a4003a00 con 0x7f74b40b17d0 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.377+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7f74a4084980 con 0x7f74b40b17d0 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.377+0000 7f74a2ffd640 10 client.4546.objecter ms_dispatch 0x7f74b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.377+0000 7f74a2ffd640 3 client.4546.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.377+0000 7f74a2ffd640 3 client.4546.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.377+0000 7f74a2ffd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.378+0000 7f74b1d9b640 10 monclient: get_auth_request con 0x7f749406df80 auth_method 0 2024-03-29T10:49:16.363 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.379+0000 7f74a2ffd640 10 client.4546.objecter ms_handle_connect 0x7f749406df80 2024-03-29T10:49:16.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.384+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74a4050090 con 0x7f74b40b17d0 2024-03-29T10:49:16.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.384+0000 7f74a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:16.366 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.384+0000 7f74a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:16.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.699+0000 7f74b898e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:16.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.699+0000 7f74b898e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:16.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.699+0000 7f74b898e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:16.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.699+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7498005330 con 0x7f74b40b17d0 2024-03-29T10:49:16.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.700+0000 7f74a2ffd640 1 -- 172.21.15.94:0/13918342 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7f74a4052190 con 0x7f74b40b17d0 2024-03-29T10:49:16.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.700+0000 7f74a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:16.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.700+0000 7f74a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:16.682 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:16.683 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 10 librados: watch_flush enter 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 10 librados: watch_flush exit 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 10 client.4546.objecter successfully canceled tick 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 >> v1:172.21.15.94:6816/3829934808 conn(0x7f749406df80 legacy=0x7f7494070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 10 monclient: shutdown 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 >> v1:172.21.15.173:6789/0 conn(0x7f74b40b17d0 legacy=0x7f74b40b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 shutdown_connections 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 >> 172.21.15.94:0/13918342 conn(0x7f74b4066a00 msgr2=0x7f74b40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:16.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.704+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 shutdown_connections 2024-03-29T10:49:16.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.705+0000 7f74b898e640 1 -- 172.21.15.94:0/13918342 wait complete. 2024-03-29T10:49:16.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:16.705+0000 7f74b898e640 1 librados: shutdown 2024-03-29T10:49:16.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:23.509 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:23.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.676+0000 7f534f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:49:23.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.676+0000 7f534f59e640 10 monclient: build_initial_monmap 2024-03-29T10:49:23.658 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.676+0000 7f534f59e640 10 monclient: monmap: 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:23.677298+0000 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:23.677298+0000 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:23.659 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:23.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.678+0000 7f534f59e640 1 Processor -- start 2024-03-29T10:49:23.661 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.678+0000 7f534f59e640 1 -- start start 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.678+0000 7f534f59e640 10 monclient: init 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f53500aac30 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f5350072f20 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f53500a9e90 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500ad8e0 con 0x7f53500aac30 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500ac6a0 con 0x7f5350072f20 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53500ab230 con 0x7f53500a9e90 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:23.662 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.679+0000 7f534f59e640 10 monclient(hunting): authenticate will time out at 1414.947798s 2024-03-29T10:49:23.665 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.683+0000 7f534ed9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f53500a9e90 0x7f53500ad1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:37831/0 (socket says 172.21.15.94:37831) 2024-03-29T10:49:23.665 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.683+0000 7f534ed9d640 1 -- 172.21.15.94:0/3793149614 learned_addr learned my addr 172.21.15.94:0/3793149614 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639807540 0 0) 0x7f53500ab230 con 0x7f53500a9e90 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5338003520 con 0x7f53500a9e90 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3856693981 0 0) 0x7f5338003520 con 0x7f53500a9e90 2024-03-29T10:49:23.667 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 >> v1:172.21.15.173:6789/0 conn(0x7f5350072f20 legacy=0x7f53500acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 >> v1:172.21.15.94:6789/0 conn(0x7f53500aac30 legacy=0x7f535013af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 1 monclient: found mon.noname-b 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.685+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535013b660 con 0x7f53500a9e90 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 10 monclient: _finish_auth 0 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5348001a10 con 0x7f53500a9e90 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 10 monclient: dump: 2024-03-29T10:49:23.668 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534f59e640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:23.669 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 1 -- 172.21.15.94:0/3793149614 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5348001ca0 con 0x7f53500a9e90 2024-03-29T10:49:23.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:23.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:23.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.686+0000 7f534f59e640 4 set_mon_vals no callback set 2024-03-29T10:49:23.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.687+0000 7f534f59e640 10 monclient: shutdown 2024-03-29T10:49:23.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.687+0000 7f534f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:23.670 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.687+0000 7f534f59e640 1 -- 172.21.15.94:0/3793149614 >> v1:172.21.15.94:6790/0 conn(0x7f53500a9e90 legacy=0x7f53500ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:23.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.687+0000 7f534f59e640 1 -- 172.21.15.94:0/3793149614 shutdown_connections 2024-03-29T10:49:23.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.687+0000 7f534f59e640 1 -- 172.21.15.94:0/3793149614 >> 172.21.15.94:0/3793149614 conn(0x7f53500b3a30 msgr2=0x7f53500b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:23.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.689+0000 7f534f59e640 1 -- 172.21.15.94:0/3793149614 shutdown_connections 2024-03-29T10:49:23.671 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.690+0000 7f534f59e640 1 -- 172.21.15.94:0/3793149614 wait complete. 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 10 monclient: build_initial_monmap 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 10 monclient: monmap: 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:23.677 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 1 librados: starting msgr at 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 1 librados: starting objecter 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 1 Processor -- start 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 1 -- start start 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 1 librados: setting wanted keys 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 1 librados: calling monclient init 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.691+0000 7f534f59e640 10 monclient: init 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f53500a9e90 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f535013d380 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f53501db970 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:23.678 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535012ec50 con 0x7f535013d380 2024-03-29T10:49:23.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535012f6a0 con 0x7f53501db970 2024-03-29T10:49:23.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535012f910 con 0x7f53500a9e90 2024-03-29T10:49:23.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:23.679 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.692+0000 7f534f59e640 10 monclient(hunting): authenticate will time out at 1414.960735s 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.695+0000 7f534ed9d640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f53501db970 0x7f53501ddd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:34669/0 (socket says 172.21.15.94:34669) 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.696+0000 7f534ed9d640 1 -- 172.21.15.94:0/4050918862 learned_addr learned my addr 172.21.15.94:0/4050918862 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.696+0000 7f533effd640 10 client.?.objecter ms_handle_connect 0x7f53501db970 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.696+0000 7f533effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.697+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326678055 0 0) 0x7f535012f6a0 con 0x7f53501db970 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.697+0000 7f533effd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.697+0000 7f533effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.697+0000 7f533effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:23.681 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.697+0000 7f533effd640 10 monclient(hunting): my global_id is 4549 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5330004eb0 con 0x7f53501db970 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584078666 0 0) 0x7f5330004eb0 con 0x7f53501db970 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5330005300 con 0x7f53501db970 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5348004d00 con 0x7f53501db970 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 10 monclient(hunting): dump: 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:23.682 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:23.683 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:23.683 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:23.683 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:23.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.698+0000 7f533effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:23.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.699+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2474052415 0 0) 0x7f5330005300 con 0x7f53501db970 2024-03-29T10:49:23.683 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.699+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 >> v1:172.21.15.94:6790/0 conn(0x7f53500a9e90 legacy=0x7f535012e410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 >> v1:172.21.15.94:6789/0 conn(0x7f535013d380 legacy=0x7f535013d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 1 monclient: found mon.b 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535012fbf0 con 0x7f53501db970 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 10 monclient: _finish_auth 0 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f533effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 5 monclient: authenticate success, global_id 4549 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 10 monclient: _renew_subs 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:23.686 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535013df40 con 0x7f53501db970 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 10 monclient: _renew_subs 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f535012d9e0 con 0x7f53501db970 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.700+0000 7f534f59e640 1 librados: init done 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f534f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f534f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f534f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535004f730 con 0x7f53501db970 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5348001e90 con 0x7f53501db970 2024-03-29T10:49:23.687 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f533effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f5348004d00 con 0x7f53501db970 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f533effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f533effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f533effd640 10 monclient: dump: 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:23.688 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:23.689 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:23.689 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:23.689 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:23.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.702+0000 7f534cd99640 4 set_mon_vals no callback set 2024-03-29T10:49:23.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.706+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f5348005190 con 0x7f53501db970 2024-03-29T10:49:23.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.706+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7f5348084cc0 con 0x7f53501db970 2024-03-29T10:49:23.689 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.706+0000 7f533effd640 10 client.4549.objecter ms_dispatch 0x7f5350000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:23.690 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.706+0000 7f533effd640 3 client.4549.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:23.690 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.706+0000 7f533effd640 3 client.4549.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:23.690 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.706+0000 7f533effd640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-29T10:49:23.690 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.706+0000 7f534e59c640 10 monclient: get_auth_request con 0x7f533006e160 auth_method 0 2024-03-29T10:49:23.690 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.707+0000 7f533effd640 10 client.4549.objecter ms_handle_connect 0x7f533006e160 2024-03-29T10:49:23.694 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.710+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53480505a0 con 0x7f53501db970 2024-03-29T10:49:23.695 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.710+0000 7f533effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:23.695 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:23.710+0000 7f533effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:24.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.084+0000 7f534f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:24.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.084+0000 7f534f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:24.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.084+0000 7f534f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:24.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.084+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53500b3640 con 0x7f53501db970 2024-03-29T10:49:24.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.085+0000 7f533effd640 1 -- 172.21.15.94:0/4050918862 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7f53480524d0 con 0x7f53501db970 2024-03-29T10:49:24.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.085+0000 7f533effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:24.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.085+0000 7f533effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:24.067 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:24.068 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.088+0000 7f534f59e640 10 librados: watch_flush enter 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.088+0000 7f534f59e640 10 librados: watch_flush exit 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.088+0000 7f534f59e640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.088+0000 7f534f59e640 10 client.4549.objecter successfully canceled tick 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 >> v1:172.21.15.94:6816/3829934808 conn(0x7f533006e160 legacy=0x7f5330070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 10 monclient: shutdown 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 >> v1:172.21.15.173:6789/0 conn(0x7f53501db970 legacy=0x7f53501ddd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 shutdown_connections 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 >> 172.21.15.94:0/4050918862 conn(0x7f53500b3a30 msgr2=0x7f5350139080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 shutdown_connections 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 1 -- 172.21.15.94:0/4050918862 wait complete. 2024-03-29T10:49:24.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:24.089+0000 7f534f59e640 1 librados: shutdown 2024-03-29T10:49:24.087 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:30.895 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:31.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.064+0000 7f1bfd493640 10 monclient: get_monmap_and_config 2024-03-29T10:49:31.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.064+0000 7f1bfd493640 10 monclient: build_initial_monmap 2024-03-29T10:49:31.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.064+0000 7f1bfd493640 10 monclient: monmap: 2024-03-29T10:49:31.047 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:31.047 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:31.065687+0000 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:31.065687+0000 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:31.048 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:31.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.066+0000 7f1bfd493640 1 Processor -- start 2024-03-29T10:49:31.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.066+0000 7f1bfd493640 1 -- start start 2024-03-29T10:49:31.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.066+0000 7f1bfd493640 10 monclient: init 2024-03-29T10:49:31.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:31.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:31.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf80b0770 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:31.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf814a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:31.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf80ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:31.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf80aba70 con 0x7f1bf80b0770 2024-03-29T10:49:31.052 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf80aecb0 con 0x7f1bf814a700 2024-03-29T10:49:31.052 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf80af150 con 0x7f1bf80ac5a0 2024-03-29T10:49:31.052 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:31.052 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.069+0000 7f1bfd493640 10 monclient(hunting): authenticate will time out at 1422.337550s 2024-03-29T10:49:31.057 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.075+0000 7f1beffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f1bf814a700 0x7f1bf80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:43647/0 (socket says 172.21.15.94:43647) 2024-03-29T10:49:31.057 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.075+0000 7f1beffff640 1 -- 172.21.15.94:0/976182288 learned_addr learned my addr 172.21.15.94:0/976182288 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:31.057 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.076+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688543139 0 0) 0x7f1bf80aecb0 con 0x7f1bf814a700 2024-03-29T10:49:31.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.076+0000 7f1bf67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.076+0000 7f1bf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.076+0000 7f1bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.076+0000 7f1bf67fc640 10 monclient(hunting): my global_id is 4552 2024-03-29T10:49:31.058 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.076+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be4003520 con 0x7f1bf814a700 2024-03-29T10:49:31.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948918440 0 0) 0x7f1bf80aba70 con 0x7f1bf80b0770 2024-03-29T10:49:31.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient(hunting): my global_id is 4581 2024-03-29T10:49:31.059 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf80aecb0 con 0x7f1bf80b0770 2024-03-29T10:49:31.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3717393517 0 0) 0x7f1bf80aecb0 con 0x7f1bf80b0770 2024-03-29T10:49:31.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 >> v1:172.21.15.94:6790/0 conn(0x7f1bf80ac5a0 legacy=0x7f1bf8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 >> v1:172.21.15.173:6789/0 conn(0x7f1bf814a700 legacy=0x7f1bf80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:31.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 monclient: found mon.noname-a 2024-03-29T10:49:31.061 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf80ac9d0 con 0x7f1bf80b0770 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1be8001670 con 0x7f1bf80b0770 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.077+0000 7f1bf67fc640 10 monclient: dump: 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.062 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.078+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1be8001850 con 0x7f1bf80b0770 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.078+0000 7f1bf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.078+0000 7f1bf67fc640 1 -- 172.21.15.94:0/976182288 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1be8001c20 con 0x7f1bf80b0770 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.078+0000 7f1bf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.078+0000 7f1bf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.078+0000 7f1bf67fc640 10 monclient: dump: 2024-03-29T10:49:31.063 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:31.064 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:31.066 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.079+0000 7f1bfd493640 5 monclient: authenticate success, global_id 4581 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.079+0000 7f1bfd493640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.079+0000 7f1bfd493640 4 set_mon_vals no callback set 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.079+0000 7f1bfd493640 10 monclient: shutdown 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.079+0000 7f1bfd493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.079+0000 7f1bfd493640 1 -- 172.21.15.94:0/976182288 >> v1:172.21.15.94:6789/0 conn(0x7f1bf80b0770 legacy=0x7f1bf8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.080+0000 7f1bfd493640 1 -- 172.21.15.94:0/976182288 shutdown_connections 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.080+0000 7f1bfd493640 1 -- 172.21.15.94:0/976182288 >> 172.21.15.94:0/976182288 conn(0x7f1bf806b660 msgr2=0x7f1bf806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.081+0000 7f1bfd493640 1 -- 172.21.15.94:0/976182288 shutdown_connections 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.081+0000 7f1bfd493640 1 -- 172.21.15.94:0/976182288 wait complete. 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 10 monclient: build_initial_monmap 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 10 monclient: monmap: 2024-03-29T10:49:31.067 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 1 librados: starting msgr at 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 1 librados: starting objecter 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 1 Processor -- start 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 1 -- start start 2024-03-29T10:49:31.068 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 1 librados: setting wanted keys 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 1 librados: calling monclient init 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.082+0000 7f1bfd493640 10 monclient: init 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf80ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf80b0770 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf814a700 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf8145b90 con 0x7f1bf814a700 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf81db450 con 0x7f1bf80b0770 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf8125d10 con 0x7f1bf80ac5a0 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:31.069 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.083+0000 7f1bfd493640 10 monclient(hunting): authenticate will time out at 1422.351601s 2024-03-29T10:49:31.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.086+0000 7f1beffff640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f1bf80b0770 0x7f1bf81db970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:43733/0 (socket says 172.21.15.94:43733) 2024-03-29T10:49:31.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.088+0000 7f1beffff640 1 -- 172.21.15.94:0/1528348632 learned_addr learned my addr 172.21.15.94:0/1528348632 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:31.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.088+0000 7f1bef7fe640 10 client.?.objecter ms_handle_connect 0x7f1bf80b0770 2024-03-29T10:49:31.070 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.088+0000 7f1bef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:31.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.089+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984333767 0 0) 0x7f1bf81db450 con 0x7f1bf80b0770 2024-03-29T10:49:31.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.089+0000 7f1bef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.089+0000 7f1bef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.089+0000 7f1bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.089+0000 7f1bef7fe640 10 monclient(hunting): my global_id is 4555 2024-03-29T10:49:31.071 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.089+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd8004eb0 con 0x7f1bf80b0770 2024-03-29T10:49:31.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.090+0000 7f1bef7fe640 10 client.?.objecter ms_handle_connect 0x7f1bf80ac5a0 2024-03-29T10:49:31.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.090+0000 7f1bef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:31.072 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.090+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093466503 0 0) 0x7f1bf8125d10 con 0x7f1bf80ac5a0 2024-03-29T10:49:31.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.090+0000 7f1bef7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:31.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:31.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:31.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:49:31.073 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf81db450 con 0x7f1bf80ac5a0 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434053747 0 0) 0x7f1bd8004eb0 con 0x7f1bf80b0770 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bf8125d10 con 0x7f1bf80b0770 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1115972435 0 0) 0x7f1bf81db450 con 0x7f1bf80ac5a0 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bd8004eb0 con 0x7f1bf80ac5a0 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.091+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1be00017d0 con 0x7f1bf80b0770 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient(hunting): dump: 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:31.074 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:31.075 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.075 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.075 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.075 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:31.075 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:31.075 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:31.075 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1be8001b60 con 0x7f1bf80ac5a0 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient(hunting): dump: 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.077 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.078 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.078 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:31.078 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:31.078 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:31.078 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:31.078 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:31.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 788690366 0 0) 0x7f1bf8125d10 con 0x7f1bf80b0770 2024-03-29T10:49:31.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 >> v1:172.21.15.94:6790/0 conn(0x7f1bf80ac5a0 legacy=0x7f1bf8145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.079 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 >> v1:172.21.15.94:6789/0 conn(0x7f1bf814a700 legacy=0x7f1bf81267f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.092+0000 7f1bef7fe640 1 monclient: found mon.b 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf81239d0 con 0x7f1bf80b0770 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 5 monclient: authenticate success, global_id 4555 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 10 monclient: _renew_subs 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf8145750 con 0x7f1bf80b0770 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:31.080 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 10 monclient: _renew_subs 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf81dc260 con 0x7f1bf80b0770 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bfd493640 1 librados: init done 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 10 client.4555.objecter ms_handle_connect 0x7f1bf814a700 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 10 client.4555.objecter resend_mon_ops 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1be0001930 con 0x7f1bf80b0770 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.093+0000 7f1bf5ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.094+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f1be0004980 con 0x7f1bf80b0770 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.094+0000 7f1bfd493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.094+0000 7f1bfd493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.094+0000 7f1bfd493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.094+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bdc005080 con 0x7f1bf80b0770 2024-03-29T10:49:31.081 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.095+0000 7f1bef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.095+0000 7f1bef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.095+0000 7f1bef7fe640 10 monclient: dump: 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:31.082 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.095+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f1be0004ba0 con 0x7f1bf80b0770 2024-03-29T10:49:31.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.096+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7f1be0084a70 con 0x7f1bf80b0770 2024-03-29T10:49:31.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.096+0000 7f1bef7fe640 10 client.4555.objecter ms_dispatch 0x7f1bf8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:31.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.096+0000 7f1bef7fe640 3 client.4555.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:31.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.096+0000 7f1bef7fe640 3 client.4555.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:31.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.096+0000 7f1bef7fe640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-29T10:49:31.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.096+0000 7f1bf6ffd640 10 monclient: get_auth_request con 0x7f1bd806de00 auth_method 0 2024-03-29T10:49:31.083 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.097+0000 7f1bef7fe640 10 client.4555.objecter ms_handle_connect 0x7f1bd806de00 2024-03-29T10:49:31.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.104+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1be0050350 con 0x7f1bf80b0770 2024-03-29T10:49:31.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.104+0000 7f1bef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:31.085 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.104+0000 7f1bef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:31.404 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.421+0000 7f1bfd493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:31.404 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.421+0000 7f1bfd493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:31.404 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.421+0000 7f1bfd493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:31.404 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.421+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bdc005330 con 0x7f1bf80b0770 2024-03-29T10:49:31.404 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.422+0000 7f1bef7fe640 1 -- 172.21.15.94:0/1528348632 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7f1be0052280 con 0x7f1bf80b0770 2024-03-29T10:49:31.404 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.422+0000 7f1bef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:31.404 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.422+0000 7f1bef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:31.405 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:31.405 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:31.408 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 10 librados: watch_flush enter 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 10 librados: watch_flush exit 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 10 client.4555.objecter successfully canceled tick 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 >> v1:172.21.15.94:6816/3829934808 conn(0x7f1bd806de00 legacy=0x7f1bd80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 10 monclient: shutdown 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 >> v1:172.21.15.173:6789/0 conn(0x7f1bf80b0770 legacy=0x7f1bf81db970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 shutdown_connections 2024-03-29T10:49:31.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.427+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 >> 172.21.15.94:0/1528348632 conn(0x7f1bf806b660 msgr2=0x7f1bf806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:31.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.429+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 shutdown_connections 2024-03-29T10:49:31.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.429+0000 7f1bfd493640 1 -- 172.21.15.94:0/1528348632 wait complete. 2024-03-29T10:49:31.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:31.429+0000 7f1bfd493640 1 librados: shutdown 2024-03-29T10:49:31.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:38.233 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:38.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.400+0000 7ff27b0ff640 10 monclient: get_monmap_and_config 2024-03-29T10:49:38.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.400+0000 7ff27b0ff640 10 monclient: build_initial_monmap 2024-03-29T10:49:38.384 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.400+0000 7ff27b0ff640 10 monclient: monmap: 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:38.402247+0000 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:38.402247+0000 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:38.385 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.401+0000 7ff27b0ff640 1 Processor -- start 2024-03-29T10:49:38.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 1 -- start start 2024-03-29T10:49:38.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient: init 2024-03-29T10:49:38.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:38.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:38.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient(hunting): picked mon.noname-b con 0x7ff2740a9ac0 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:38.386 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient(hunting): picked mon.noname-a con 0x7ff27413ee90 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:38.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient(hunting): picked mon.noname-c con 0x7ff2740b4100 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:38.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2740af6f0 con 0x7ff27413ee90 2024-03-29T10:49:38.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2740aac80 con 0x7ff2740b4100 2024-03-29T10:49:38.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2740aaf60 con 0x7ff2740a9ac0 2024-03-29T10:49:38.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:38.387 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.402+0000 7ff27b0ff640 10 monclient(hunting): authenticate will time out at 1429.672068s 2024-03-29T10:49:38.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.405+0000 7ff27969c640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff2740b4100 0x7ff27414a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.94:45174/0 (socket says 172.21.15.94:45174) 2024-03-29T10:49:38.388 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.405+0000 7ff27969c640 1 -- 172.21.15.94:0/373479711 learned_addr learned my addr 172.21.15.94:0/373479711 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374487306 0 0) 0x7ff2740af6f0 con 0x7ff27413ee90 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25c003520 con 0x7ff27413ee90 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112085151 0 0) 0x7ff2740aac80 con 0x7ff2740b4100 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): my global_id is 4558 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2740af6f0 con 0x7ff2740b4100 2024-03-29T10:49:38.392 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675023033 0 0) 0x7ff2740aaf60 con 0x7ff2740a9ac0 2024-03-29T10:49:38.393 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:38.393 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:38.393 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:38.393 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:49:38.393 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2740aac80 con 0x7ff2740a9ac0 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.409+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727311522 0 0) 0x7ff2740aac80 con 0x7ff2740a9ac0 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 >> v1:172.21.15.173:6789/0 conn(0x7ff2740b4100 legacy=0x7ff27414a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 >> v1:172.21.15.94:6789/0 conn(0x7ff27413ee90 legacy=0x7ff27413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 1 monclient: found mon.noname-b 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.94:6790/0 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27413e970 con 0x7ff2740a9ac0 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff264001530 con 0x7ff2740a9ac0 2024-03-29T10:49:38.394 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: dump: 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff27b0ff640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:49:38.395 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff27b0ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:38.397 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 <== mon.2 v1:172.21.15.94:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff264001750 con 0x7ff2740a9ac0 2024-03-29T10:49:38.397 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:38.397 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff27b0ff640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:38.397 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff27b0ff640 4 set_mon_vals no callback set 2024-03-29T10:49:38.397 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 1 -- 172.21.15.94:0/373479711 <== mon.2 v1:172.21.15.94:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff264001ae0 con 0x7ff2740a9ac0 2024-03-29T10:49:38.397 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.397 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.410+0000 7ff2737fe640 10 monclient: dump: 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.411+0000 7ff27b0ff640 10 monclient: shutdown 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.411+0000 7ff27b0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.411+0000 7ff27b0ff640 1 -- 172.21.15.94:0/373479711 >> v1:172.21.15.94:6790/0 conn(0x7ff2740a9ac0 legacy=0x7ff2740adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.398 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.413+0000 7ff27b0ff640 1 -- 172.21.15.94:0/373479711 shutdown_connections 2024-03-29T10:49:38.399 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.413+0000 7ff27b0ff640 1 -- 172.21.15.94:0/373479711 >> 172.21.15.94:0/373479711 conn(0x7ff2740b35e0 msgr2=0x7ff2740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:38.399 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.414+0000 7ff27b0ff640 1 -- 172.21.15.94:0/373479711 shutdown_connections 2024-03-29T10:49:38.399 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.414+0000 7ff27b0ff640 1 -- 172.21.15.94:0/373479711 wait complete. 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.415+0000 7ff27b0ff640 10 monclient: build_initial_monmap 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.415+0000 7ff27b0ff640 10 monclient: monmap: 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:38.400 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.415+0000 7ff27b0ff640 1 librados: starting msgr at 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.415+0000 7ff27b0ff640 1 librados: starting objecter 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.416+0000 7ff27b0ff640 1 Processor -- start 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.416+0000 7ff27b0ff640 1 -- start start 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.416+0000 7ff27b0ff640 1 librados: setting wanted keys 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.416+0000 7ff27b0ff640 1 librados: calling monclient init 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.416+0000 7ff27b0ff640 10 monclient: init 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.417+0000 7ff27b0ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.417+0000 7ff27b0ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:38.401 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.417+0000 7ff27b0ff640 10 monclient(hunting): picked mon.noname-a con 0x7ff2740a9ac0 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:38.408 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.417+0000 7ff27b0ff640 10 monclient(hunting): picked mon.noname-b con 0x7ff2740b4100 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:38.408 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.417+0000 7ff27b0ff640 10 monclient(hunting): picked mon.noname-c con 0x7ff27413ee90 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:38.408 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.417+0000 7ff27b0ff640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff274141db0 con 0x7ff2740a9ac0 2024-03-29T10:49:38.408 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.420+0000 7ff27969c640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7ff27413ee90 0x7ff27414e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:42175/0 (socket says 172.21.15.94:42175) 2024-03-29T10:49:38.408 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.420+0000 7ff27969c640 1 -- 172.21.15.94:0/1782048431 learned_addr learned my addr 172.21.15.94:0/1782048431 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27414e780 con 0x7ff2740b4100 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27414fe40 con 0x7ff27413ee90 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff27b0ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff27b0ff640 10 monclient(hunting): authenticate will time out at 1429.690363s 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 client.?.objecter ms_handle_connect 0x7ff27413ee90 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 client.?.objecter ms_handle_connect 0x7ff2740b4100 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170989905 0 0) 0x7ff27414e780 con 0x7ff2740b4100 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 10 monclient(hunting): my global_id is 4561 2024-03-29T10:49:38.409 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.421+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff260004eb0 con 0x7ff2740b4100 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032286048 0 0) 0x7ff27414fe40 con 0x7ff27413ee90 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff27414e780 con 0x7ff27413ee90 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2537101948 0 0) 0x7ff260004eb0 con 0x7ff2740b4100 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff27414fe40 con 0x7ff2740b4100 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff268001890 con 0x7ff2740b4100 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): dump: 2024-03-29T10:49:38.410 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.422+0000 7ff2717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 10 client.?.objecter ms_handle_connect 0x7ff2740a9ac0 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1879848572 0 0) 0x7ff27414fe40 con 0x7ff2740b4100 2024-03-29T10:49:38.411 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 >> v1:172.21.15.94:6790/0 conn(0x7ff27413ee90 legacy=0x7ff27414e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 >> v1:172.21.15.94:6789/0 conn(0x7ff2740a9ac0 legacy=0x7ff27414ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 1 monclient: found mon.b 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff274150120 con 0x7ff2740b4100 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2680035f0 con 0x7ff2740b4100 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.423+0000 7ff2717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7ff268003900 con 0x7ff2740b4100 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff272ffd640 4 set_mon_vals no callback set 2024-03-29T10:49:38.412 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 5 monclient: authenticate success, global_id 4561 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 10 monclient: _renew_subs 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff274150400 con 0x7ff2740b4100 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 10 monclient: _renew_subs 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.424+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff274123bb0 con 0x7ff2740b4100 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.425+0000 7ff27b0ff640 1 librados: init done 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.425+0000 7ff2717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.425+0000 7ff2717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.425+0000 7ff2717fa640 10 monclient: dump: 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.413 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:38.414 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:38.414 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:38.414 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:38.414 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:38.414 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:38.414 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.427+0000 7ff27b0ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.427+0000 7ff27b0ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.427+0000 7ff27b0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.427+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff274075f10 con 0x7ff2740b4100 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.431+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7ff2680198d0 con 0x7ff2740b4100 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.432+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7ff268083560 con 0x7ff2740b4100 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.432+0000 7ff2717fa640 10 client.4561.objecter ms_dispatch 0x7ff274000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.432+0000 7ff2717fa640 3 client.4561.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.432+0000 7ff2717fa640 3 client.4561.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:38.415 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.432+0000 7ff2717fa640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-29T10:49:38.418 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.435+0000 7ff278e9b640 10 monclient: get_auth_request con 0x7ff26006de30 auth_method 0 2024-03-29T10:49:38.418 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.436+0000 7ff2717fa640 10 client.4561.objecter ms_handle_connect 0x7ff26006de30 2024-03-29T10:49:38.421 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.438+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff26804ee40 con 0x7ff2740b4100 2024-03-29T10:49:38.421 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.438+0000 7ff2717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:38.421 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.438+0000 7ff2717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:38.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.762+0000 7ff27b0ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:38.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.762+0000 7ff27b0ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:38.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.762+0000 7ff27b0ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:38.745 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.762+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff274123bb0 con 0x7ff2740b4100 2024-03-29T10:49:38.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.765+0000 7ff2717fa640 1 -- 172.21.15.94:0/1782048431 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7ff274123bb0 con 0x7ff2740b4100 2024-03-29T10:49:38.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.765+0000 7ff2717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:38.748 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.765+0000 7ff2717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:38.749 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:38.749 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 10 librados: watch_flush enter 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 10 librados: watch_flush exit 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 10 client.4561.objecter successfully canceled tick 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 >> v1:172.21.15.94:6816/3829934808 conn(0x7ff26006de30 legacy=0x7ff2600702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 10 monclient: shutdown 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:38.752 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.769+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 >> v1:172.21.15.173:6789/0 conn(0x7ff2740b4100 legacy=0x7ff27414d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:38.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.771+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 shutdown_connections 2024-03-29T10:49:38.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.771+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 >> 172.21.15.94:0/1782048431 conn(0x7ff2740b35e0 msgr2=0x7ff2740b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:38.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.771+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 shutdown_connections 2024-03-29T10:49:38.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.771+0000 7ff27b0ff640 1 -- 172.21.15.94:0/1782048431 wait complete. 2024-03-29T10:49:38.754 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:38.771+0000 7ff27b0ff640 1 librados: shutdown 2024-03-29T10:49:38.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:45.577 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:45.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.729+0000 7f4171064640 10 monclient: get_monmap_and_config 2024-03-29T10:49:45.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.730+0000 7f4171064640 10 monclient: build_initial_monmap 2024-03-29T10:49:45.712 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.730+0000 7f4171064640 10 monclient: monmap: 2024-03-29T10:49:45.712 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:45.712 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:45.712 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:45.730883+0000 2024-03-29T10:49:45.712 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:45.730883+0000 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.730+0000 7f4171064640 1 Processor -- start 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.730+0000 7f4171064640 1 -- start start 2024-03-29T10:49:45.713 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.730+0000 7f4171064640 10 monclient: init 2024-03-29T10:49:45.714 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.731+0000 7f4171064640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:45.714 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.731+0000 7f4171064640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.731+0000 7f4171064640 10 monclient(hunting): picked mon.noname-b con 0x7f416c0b0770 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f4171064640 10 monclient(hunting): picked mon.noname-c con 0x7f416c14a700 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f4171064640 10 monclient(hunting): picked mon.noname-a con 0x7f416c0ac5a0 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f4171064640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0aba70 con 0x7f416c0ac5a0 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f4171064640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0aecb0 con 0x7f416c14a700 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f4171064640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0af150 con 0x7f416c0b0770 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f4171064640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f4171064640 10 monclient(hunting): authenticate will time out at 1437.000521s 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f416b59e640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f416c0ac5a0 0x7f416c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:49632/0 (socket says 172.21.15.94:49632) 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.732+0000 7f416b59e640 1 -- 172.21.15.94:0/618878097 learned_addr learned my addr 172.21.15.94:0/618878097 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.733+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056170667 0 0) 0x7f416c0aba70 con 0x7f416c0ac5a0 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.733+0000 7f4169d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.733+0000 7f4169d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.733+0000 7f4169d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:45.715 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.733+0000 7f4169d9b640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:49:45.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.733+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4154003520 con 0x7f416c0ac5a0 2024-03-29T10:49:45.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.734+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1214820270 0 0) 0x7f4154003520 con 0x7f416c0ac5a0 2024-03-29T10:49:45.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.734+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 >> v1:172.21.15.94:6790/0 conn(0x7f416c0b0770 legacy=0x7f416c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:45.716 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.734+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 >> v1:172.21.15.173:6789/0 conn(0x7f416c14a700 legacy=0x7f416c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 1 monclient: found mon.noname-a 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.94:6789/0 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0ac9d0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: _finish_auth 0 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4164001a10 con 0x7f416c0ac5a0 2024-03-29T10:49:45.717 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: dump: 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4171064640 5 monclient: authenticate success, global_id 4593 2024-03-29T10:49:45.718 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4171064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 <== mon.0 v1:172.21.15.94:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4164001ca0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 1 -- 172.21.15.94:0/618878097 <== mon.0 v1:172.21.15.94:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f4164002030 con 0x7f416c0ac5a0 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4169d9b640 10 monclient: dump: 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:45.719 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:45.720 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4171064640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.735+0000 7f4171064640 4 set_mon_vals no callback set 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.736+0000 7f4171064640 10 monclient: shutdown 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.736+0000 7f4171064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.736+0000 7f4171064640 1 -- 172.21.15.94:0/618878097 >> v1:172.21.15.94:6789/0 conn(0x7f416c0ac5a0 legacy=0x7f416c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.736+0000 7f4171064640 1 -- 172.21.15.94:0/618878097 shutdown_connections 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.736+0000 7f4171064640 1 -- 172.21.15.94:0/618878097 >> 172.21.15.94:0/618878097 conn(0x7f416c06b660 msgr2=0x7f416c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:45.721 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.737+0000 7f4171064640 1 -- 172.21.15.94:0/618878097 shutdown_connections 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.737+0000 7f4171064640 1 -- 172.21.15.94:0/618878097 wait complete. 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.738+0000 7f4171064640 10 monclient: build_initial_monmap 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.738+0000 7f4171064640 10 monclient: monmap: 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:45.722 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.739+0000 7f4171064640 1 librados: starting msgr at 2024-03-29T10:49:45.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.739+0000 7f4171064640 1 librados: starting objecter 2024-03-29T10:49:45.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.739+0000 7f4171064640 1 Processor -- start 2024-03-29T10:49:45.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.739+0000 7f4171064640 1 -- start start 2024-03-29T10:49:45.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.739+0000 7f4171064640 1 librados: setting wanted keys 2024-03-29T10:49:45.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.739+0000 7f4171064640 1 librados: calling monclient init 2024-03-29T10:49:45.723 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.739+0000 7f4171064640 10 monclient: init 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.740+0000 7f4171064640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.740+0000 7f4171064640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.740+0000 7f4171064640 10 monclient(hunting): picked mon.noname-c con 0x7f416c0ac5a0 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.740+0000 7f416ad9d640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f416c0ac5a0 0x7f416c1267b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:46104/0 (socket says 172.21.15.94:46104) 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.740+0000 7f416ad9d640 1 -- 172.21.15.94:0/3374717914 learned_addr learned my addr 172.21.15.94:0/3374717914 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f4171064640 10 monclient(hunting): picked mon.noname-b con 0x7f416c0b0770 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f41537fe640 10 client.?.objecter ms_handle_connect 0x7f416c0ac5a0 2024-03-29T10:49:45.730 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f41537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f4171064640 10 monclient(hunting): picked mon.noname-a con 0x7f416c14a700 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f4171064640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c126320 con 0x7f416c14a700 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f4171064640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c128310 con 0x7f416c0b0770 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f4171064640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c125eb0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.741+0000 7f4171064640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f4171064640 10 monclient(hunting): authenticate will time out at 1437.010116s 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090415870 0 0) 0x7f416c125eb0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4158004eb0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 client.?.objecter ms_handle_connect 0x7f416c14a700 2024-03-29T10:49:45.731 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501235432 0 0) 0x7f416c126320 con 0x7f416c14a700 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.742+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f416c125eb0 con 0x7f416c14a700 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542110818 0 0) 0x7f4158004eb0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f416c126320 con 0x7f416c0ac5a0 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f415c001a30 con 0x7f416c0ac5a0 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient(hunting): dump: 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:45.732 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2655931078 0 0) 0x7f416c126320 con 0x7f416c0ac5a0 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 >> v1:172.21.15.173:6789/0 conn(0x7f416c0b0770 legacy=0x7f416c144e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 >> v1:172.21.15.94:6789/0 conn(0x7f416c14a700 legacy=0x7f416c1e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 monclient: found mon.c 2024-03-29T10:49:45.733 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c127310 con 0x7f416c0ac5a0 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f41537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 5 monclient: authenticate success, global_id 4652 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 10 monclient: _renew_subs 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c1239d0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 10 monclient: _renew_subs 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:45.734 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c1270a0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.743+0000 7f4171064640 1 librados: init done 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.745+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f415c001b90 con 0x7f416c0ac5a0 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.745+0000 7f41537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.745+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f415c003a60 con 0x7f416c0ac5a0 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.745+0000 7f41537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.745+0000 7f41537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.745+0000 7f41537fe640 10 monclient: dump: 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:45.735 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.745+0000 7f416959a640 4 set_mon_vals no callback set 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.746+0000 7f41517fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.746+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f415c003c80 con 0x7f416c0ac5a0 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.746+0000 7f41517fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.746+0000 7f41517fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.746+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414c005080 con 0x7f416c0ac5a0 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.747+0000 7f416a59c640 10 monclient: get_auth_request con 0x7f415806dad0 auth_method 0 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.747+0000 7f41537fe640 10 client.4652.objecter ms_handle_connect 0x7f415806dad0 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.747+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7f415c084b50 con 0x7f416c0ac5a0 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.747+0000 7f41537fe640 10 client.4652.objecter ms_dispatch 0x7f416c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:45.736 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.747+0000 7f41537fe640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:45.737 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.747+0000 7f41537fe640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:45.737 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.747+0000 7f41537fe640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T10:49:45.737 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.754+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f415c0505b0 con 0x7f416c0ac5a0 2024-03-29T10:49:45.737 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.754+0000 7f41537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:45.737 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:45.755+0000 7f41537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:46.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.069+0000 7f41517fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:46.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.069+0000 7f41517fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:46.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.069+0000 7f41517fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.94:6790/0 2024-03-29T10:49:46.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.069+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 --> v1:172.21.15.94:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f414c005330 con 0x7f416c0ac5a0 2024-03-29T10:49:46.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.069+0000 7f41537fe640 1 -- 172.21.15.94:0/3374717914 <== mon.2 v1:172.21.15.94:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7f415c052420 con 0x7f416c0ac5a0 2024-03-29T10:49:46.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.069+0000 7f41537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:46.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.069+0000 7f41537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:46.052 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:46.052 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:46.054 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 10 librados: watch_flush enter 2024-03-29T10:49:46.054 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 10 librados: watch_flush exit 2024-03-29T10:49:46.054 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T10:49:46.054 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 10 client.4652.objecter successfully canceled tick 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 >> v1:172.21.15.94:6816/3829934808 conn(0x7f415806dad0 legacy=0x7f415806ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 10 monclient: shutdown 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.072+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 >> v1:172.21.15.94:6790/0 conn(0x7f416c0ac5a0 legacy=0x7f416c1267b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.073+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 shutdown_connections 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.073+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 >> 172.21.15.94:0/3374717914 conn(0x7f416c06b660 msgr2=0x7f416c0ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.073+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 shutdown_connections 2024-03-29T10:49:46.055 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.074+0000 7f41517fa640 1 -- 172.21.15.94:0/3374717914 wait complete. 2024-03-29T10:49:46.056 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:46.074+0000 7f41517fa640 1 librados: shutdown 2024-03-29T10:49:46.070 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:52.876 DEBUG:teuthology.orchestra.run.smithi094:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:53.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.042+0000 7f35ee7d1640 10 monclient: get_monmap_and_config 2024-03-29T10:49:53.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.042+0000 7f35ee7d1640 10 monclient: build_initial_monmap 2024-03-29T10:49:53.024 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.042+0000 7f35ee7d1640 10 monclient: monmap: 2024-03-29T10:49:53.024 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:49:53.043799+0000 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:49:53.043799+0000 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-c 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-b 2024-03-29T10:49:53.025 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.044+0000 7f35ee7d1640 1 Processor -- start 2024-03-29T10:49:53.026 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.044+0000 7f35ee7d1640 1 -- start start 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.044+0000 7f35ee7d1640 10 monclient: init 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 10 monclient(hunting): picked mon.noname-c con 0x7f35e0006ec0 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 10 monclient(hunting): picked mon.noname-b con 0x7f35e0033530 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 10 monclient(hunting): picked mon.noname-a con 0x7f35e00cfcb0 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00cf110 con 0x7f35e00cfcb0 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e002c1d0 con 0x7f35e0006ec0 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00cbb90 con 0x7f35e0033530 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:53.027 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35ee7d1640 10 monclient(hunting): authenticate will time out at 1444.313248s 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35edfd0640 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f35e00cfcb0 0x7f35e00d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33106/0 (socket says 172.21.15.94:33106) 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.045+0000 7f35edfd0640 1 -- 172.21.15.94:0/2137143005 learned_addr learned my addr 172.21.15.94:0/2137143005 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397597687 0 0) 0x7f35e002c1d0 con 0x7f35e0006ec0 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient(hunting): my global_id is 4564 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d4003520 con 0x7f35e0006ec0 2024-03-29T10:49:53.029 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3070300943 0 0) 0x7f35d4003520 con 0x7f35e0006ec0 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 >> v1:172.21.15.94:6790/0 conn(0x7f35e0033530 legacy=0x7f35e00cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 >> v1:172.21.15.94:6789/0 conn(0x7f35e00cfcb0 legacy=0x7f35e00d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 monclient: found mon.noname-c 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.173:6789/0 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e0029c30 con 0x7f35e0006ec0 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: _finish_auth 0 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35ee7d1640 5 monclient: authenticate success, global_id 4564 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35ee7d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f35e4001670 con 0x7f35e0006ec0 2024-03-29T10:49:53.030 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: dump: 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35ee7d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:53.031 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e4001900 con 0x7f35e0006ec0 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35ee7d1640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35ee7d1640 4 set_mon_vals no callback set 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.046+0000 7f35dffff640 1 -- 172.21.15.94:0/2137143005 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f35e4001cd0 con 0x7f35e0006ec0 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35dffff640 10 monclient: dump: 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.032 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35ee7d1640 10 monclient: shutdown 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35ee7d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35ee7d1640 1 -- 172.21.15.94:0/2137143005 >> v1:172.21.15.173:6789/0 conn(0x7f35e0006ec0 legacy=0x7f35e0032e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35ee7d1640 1 -- 172.21.15.94:0/2137143005 shutdown_connections 2024-03-29T10:49:53.033 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35ee7d1640 1 -- 172.21.15.94:0/2137143005 >> 172.21.15.94:0/2137143005 conn(0x7f35e0005d40 msgr2=0x7f35e0006150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35ee7d1640 1 -- 172.21.15.94:0/2137143005 shutdown_connections 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.047+0000 7f35ee7d1640 1 -- 172.21.15.94:0/2137143005 wait complete. 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.048+0000 7f35ee7d1640 10 monclient: build_initial_monmap 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.048+0000 7f35ee7d1640 10 monclient: monmap: 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:epoch 0 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:last_changed 0.000000 2024-03-29T10:49:53.040 INFO:teuthology.misc.health.smithi094.stderr:created 0.000000 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.noname-a 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.noname-b 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.noname-c 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.049+0000 7f35ee7d1640 1 librados: starting msgr at 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.049+0000 7f35ee7d1640 1 librados: starting objecter 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.049+0000 7f35ee7d1640 1 Processor -- start 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.049+0000 7f35ee7d1640 1 -- start start 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.049+0000 7f35ee7d1640 1 librados: setting wanted keys 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.049+0000 7f35ee7d1640 1 librados: calling monclient init 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.049+0000 7f35ee7d1640 10 monclient: init 2024-03-29T10:49:53.041 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 10 monclient(hunting): picked mon.noname-b con 0x7f35e0006ec0 addr v1:172.21.15.173:6789/0 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 10 monclient(hunting): picked mon.noname-a con 0x7f35e00cfcb0 addr v1:172.21.15.94:6789/0 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 10 monclient(hunting): picked mon.noname-c con 0x7f35e002dbd0 addr v1:172.21.15.94:6790/0 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e0032940 con 0x7f35e00cfcb0 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e0032bb0 con 0x7f35e0006ec0 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 1 -- --> v1:172.21.15.94:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e0164390 con 0x7f35e002dbd0 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ee7d1640 10 monclient(hunting): authenticate will time out at 1444.318272s 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35edfd0640 1 --1- >> v1:172.21.15.94:6790/0 conn(0x7f35e002dbd0 0x7f35e016c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.94:6790/0 says I am v1:172.21.15.94:60938/0 (socket says 172.21.15.94:60938) 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35edfd0640 1 -- 172.21.15.94:0/4011826186 learned_addr learned my addr 172.21.15.94:0/4011826186 (peer_addr_for_me v1:172.21.15.94:0/0) 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 client.?.objecter ms_handle_connect 0x7f35e00cfcb0 2024-03-29T10:49:53.042 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 client.?.objecter ms_handle_connect 0x7f35e002dbd0 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 client.?.objecter ms_handle_connect 0x7f35e0006ec0 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.2 v1:172.21.15.94:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954255602 0 0) 0x7f35e0164390 con 0x7f35e002dbd0 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.050+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.94:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35cc004eb0 con 0x7f35e002dbd0 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.0 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636264507 0 0) 0x7f35e0032940 con 0x7f35e00cfcb0 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.043 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4602 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e0164390 con 0x7f35e00cfcb0 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186447658 0 0) 0x7f35e0032bb0 con 0x7f35e0006ec0 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4567 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e0032940 con 0x7f35e0006ec0 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014599112 0 0) 0x7f35e0032940 con 0x7f35e0006ec0 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e0032bb0 con 0x7f35e0006ec0 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f35e4001cd0 con 0x7f35e0006ec0 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): dump: 2024-03-29T10:49:53.044 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.0 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2393764118 0 0) 0x7f35e0164390 con 0x7f35e00cfcb0 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.94:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e0032940 con 0x7f35e00cfcb0 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.0 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f35e804fdd0 con 0x7f35e00cfcb0 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.045 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): dump: 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.2 v1:172.21.15.94:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3629831633 0 0) 0x7f35cc004eb0 con 0x7f35e002dbd0 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.94:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35e0164390 con 0x7f35e002dbd0 2024-03-29T10:49:53.046 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.2 v1:172.21.15.94:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f35d8001790 con 0x7f35e002dbd0 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 10 monclient(hunting): dump: 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.047 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4192224096 0 0) 0x7f35e0032bb0 con 0x7f35e0006ec0 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.051+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 >> v1:172.21.15.94:6790/0 conn(0x7f35e002dbd0 legacy=0x7f35e016c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 >> v1:172.21.15.94:6789/0 conn(0x7f35e00cfcb0 legacy=0x7f35e00c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 1 monclient: found mon.b 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e0164fc0 con 0x7f35e0006ec0 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ee7d1640 5 monclient: authenticate success, global_id 4567 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ee7d1640 10 monclient: _renew_subs 2024-03-29T10:49:53.048 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ee7d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.052+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e00307c0 con 0x7f35e0006ec0 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.053+0000 7f35ee7d1640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.053+0000 7f35ee7d1640 10 monclient: _renew_subs 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.053+0000 7f35ee7d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.053+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e00c62a0 con 0x7f35e0006ec0 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.053+0000 7f35ee7d1640 1 librados: init done 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e4003960 con 0x7f35e0006ec0 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4137212783 0 0) 0x7f35e4003c70 con 0x7f35e0006ec0 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35deffd640 4 set_mon_vals no callback set 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35ddffb640 10 monclient: dump: 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:epoch 1 2024-03-29T10:49:53.049 INFO:teuthology.misc.health.smithi094.stderr:fsid 86a40404-ccd6-4767-bb34-c1deee5abdf7 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:last_changed 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:created 2024-03-29T10:38:19.481071+0000 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:election_strategy: 1 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:0: v1:172.21.15.94:6789/0 mon.a 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:1: v1:172.21.15.173:6789/0 mon.b 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:2: v1:172.21.15.94:6790/0 mon.c 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr: 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.054+0000 7f35ee7d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.055+0000 7f35ee7d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.055+0000 7f35ee7d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.055+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c8005080 con 0x7f35e0006ec0 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.056+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 477861600 0 0) 0x7f35e4019c40 con 0x7f35e0006ec0 2024-03-29T10:49:53.050 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.057+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3135702537 0 0) 0x7f35e40838e0 con 0x7f35e0006ec0 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.057+0000 7f35ddffb640 10 client.4567.objecter ms_dispatch 0x7f35e8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.057+0000 7f35ddffb640 3 client.4567.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.057+0000 7f35ddffb640 3 client.4567.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.057+0000 7f35ddffb640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.057+0000 7f35ecfce640 10 monclient: get_auth_request con 0x7f35cc06dfa0 auth_method 0 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.057+0000 7f35ddffb640 10 client.4567.objecter ms_handle_connect 0x7f35cc06dfa0 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.065+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35e404f1c0 con 0x7f35e0006ec0 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.065+0000 7f35ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:53.051 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.065+0000 7f35ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:53.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.450+0000 7f35ee7d1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:53.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.450+0000 7f35ee7d1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:53.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.450+0000 7f35ee7d1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.173:6789/0 2024-03-29T10:49:53.432 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.450+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35c8005330 con 0x7f35e0006ec0 2024-03-29T10:49:53.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.453+0000 7f35ddffb640 1 -- 172.21.15.94:0/4011826186 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2443117947) 0x7f35e40510f0 con 0x7f35e0006ec0 2024-03-29T10:49:53.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.453+0000 7f35ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:53.435 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.453+0000 7f35ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:53.435 INFO:teuthology.misc.health.smithi094.stdout: 2024-03-29T10:49:53.435 INFO:teuthology.misc.health.smithi094.stdout:{"epoch":11,"fsid":"86a40404-ccd6-4767-bb34-c1deee5abdf7","created":"2024-03-29T10:38:55.698608+0000","modified":"2024-03-29T10:49:00.684720+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:00.684720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a30ab5d7-7189-4c97-81b1-72012cadfce5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"b6136696-9783-4145-bdff-58e1bc4e42cb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"6c79b11a-25e7-429a-a953-54e94244c8fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8b191e31-05ef-44ed-ae87-a385f46165f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0ae457fd-cdc7-4cad-872a-edf9e37c4a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1979d004-656d-4403-a22c-080213311414","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a33f62a9-e415-4be0-a553-c3c6085e1d3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f6bab91-7ecd-4820-827f-9151d44958dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:53.438 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.456+0000 7f35ee7d1640 10 librados: watch_flush enter 2024-03-29T10:49:53.438 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.456+0000 7f35ee7d1640 10 librados: watch_flush exit 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.456+0000 7f35ee7d1640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.456+0000 7f35ee7d1640 10 client.4567.objecter successfully canceled tick 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 >> v1:172.21.15.94:6816/3829934808 conn(0x7f35cc06dfa0 legacy=0x7f35cc070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 10 monclient: shutdown 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 >> v1:172.21.15.173:6789/0 conn(0x7f35e0006ec0 legacy=0x7f35e01648b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 shutdown_connections 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 >> 172.21.15.94:0/4011826186 conn(0x7f35e0005d40 msgr2=0x7f35e0034bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 shutdown_connections 2024-03-29T10:49:53.439 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 1 -- 172.21.15.94:0/4011826186 wait complete. 2024-03-29T10:49:53.440 INFO:teuthology.misc.health.smithi094.stderr:2024-03-29T10:49:53.457+0000 7f35ee7d1640 1 librados: shutdown 2024-03-29T10:49:53.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:53.453 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:49:53.454 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:49:53.455 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:49:53.455 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.494 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:49:53.494 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:49:53.494 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.570 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:49:53.570 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:49:53.570 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.636 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:49:53.636 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:49:53.636 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.716 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:49:53.716 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:49:53.716 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.749 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:49:53.750 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:49:53.750 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.825 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:49:53.825 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:49:53.825 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.901 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:49:53.902 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:49:53.902 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.970 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:49:53.971 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:49:53.971 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:49:53.971 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:53.995 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:49:53.995 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:49:53.995 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:54.035 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:49:54.035 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:49:54.035 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:49:54.035 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:54.047 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:49:54.047 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:49:54.048 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:54.099 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:49:54.099 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:49:54.099 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:49:54.110 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:49:54.110 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:49:54.110 DEBUG:teuthology.orchestra.run.smithi094:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:49:54.455 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:49:54.456 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:49:59.232 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:49:59.232 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:49:59.363 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:49:59.363 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:49:59.430 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:49:59.430 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:49:59.487 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:49:59.487 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:50:00.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:50:00.231 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:50:00.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:50:00.338 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:50:00.432 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:50:00.433 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:50:00.491 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:50:00.493 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/data/mon.a.tgz 2024-03-29T10:50:00.493 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-29T10:50:00.511 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.rWo78B0W8m 2024-03-29T10:50:00.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rWo78B0W8m 2024-03-29T10:50:00.628 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.rWo78B0W8m 2024-03-29T10:50:00.721 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.rWo78B0W8m is 211KB 2024-03-29T10:50:00.789 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.rWo78B0W8m 2024-03-29T10:50:00.806 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/data/mon.c.tgz 2024-03-29T10:50:00.806 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2024-03-29T10:50:00.863 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.kTKh1HmnsC 2024-03-29T10:50:00.864 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.kTKh1HmnsC 2024-03-29T10:50:00.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.kTKh1HmnsC 2024-03-29T10:50:01.067 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.kTKh1HmnsC is 211KB 2024-03-29T10:50:01.161 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.kTKh1HmnsC 2024-03-29T10:50:01.176 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/data/mon.b.tgz 2024-03-29T10:50:01.177 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-29T10:50:01.195 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.lV2ZigzZOW 2024-03-29T10:50:01.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.lV2ZigzZOW 2024-03-29T10:50:01.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.lV2ZigzZOW 2024-03-29T10:50:01.401 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.lV2ZigzZOW is 211KB 2024-03-29T10:50:01.494 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.lV2ZigzZOW 2024-03-29T10:50:01.509 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:50:01.509 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:50:01.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:50:01.596 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:50:01.634 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:50:01.634 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:50:01.636 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/remote/smithi094/crash 2024-03-29T10:50:01.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:50:01.665 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/remote/smithi173/crash 2024-03-29T10:50:01.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:50:01.700 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:50:01.700 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:50:01.710 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:gzip/var/log/ceph/ceph-client.admin.40144.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38583.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38817.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38583.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39676.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38463.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38583.log.gz 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40615.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41630.log 2024-03-29T10:50:01.774 INFO:teuthology.orchestra.run.smithi094.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39676.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41396.log 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38463.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40615.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40888.log 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41630.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41396.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39051.log 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41044.log 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41357.log 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40888.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-03-29T10:50:01.775 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40339.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41044.log.gz 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40339.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41357.log.gz 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40771.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40339.log.gz 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38739.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40771.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40693.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40420.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40771.log.gz 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38268.log 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2024-03-29T10:50:01.776 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40420.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40693.log.gz 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40420.log.gz 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38661.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38268.log.gz 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39246.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38661.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38661.log.gz 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.audit.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39246.log.gz 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-03-29T10:50:01.777 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39910.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38973.log 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38622.log 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39363.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40732.log 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38622.log: 86.2%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.38973.log.gz 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41474.log 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38622.log.gz 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40732.log: /var/log/ceph/ceph-client.admin.41474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40732.log.gz 2024-03-29T10:50:01.778 INFO:teuthology.orchestra.run.smithi094.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41474.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40183.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40537.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40576.log 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40576.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28719.log 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40027.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40576.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.tmp-client.admin.28719.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28719.log.gz 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr:gzip 85.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz -- 2024-03-29T10:50:01.779 INFO:teuthology.orchestra.run.smithi094.stderr: /var/log/ceph/ceph-client.admin.38856.log 2024-03-29T10:50:01.780 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2024-03-29T10:50:01.780 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-03-29T10:50:01.780 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40222.log: gzip 86.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.38856.log.gz --verbose 2024-03-29T10:50:01.780 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-client.admin.38385.log 2024-03-29T10:50:01.780 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39480.log: gzip -5 --verbose -- 86.4% /var/log/ceph/ceph-client.admin.39832.log 2024-03-29T10:50:01.780 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2024-03-29T10:50:01.780 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34829.log 2024-03-29T10:50:01.781 INFO:teuthology.orchestra.run.smithi094.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2024-03-29T10:50:01.781 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39832.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40927.log 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34829.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34829.log.gz 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40927.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39754.log 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39402.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40927.log.gz 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41552.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39754.log.gz 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40654.log 2024-03-29T10:50:01.782 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38700.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38229.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38700.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40654.log.gz 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39012.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38700.log.gz 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38424.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38544.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38424.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39012.log.gz 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41513.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39207.log 2024-03-29T10:50:01.783 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38544.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38424.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38544.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40849.log 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.41513.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39207.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39988.log 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38307.log 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40849.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39988.log: gzip -5 --verbose 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38307.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39988.log.gz 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38934.log 2024-03-29T10:50:01.784 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mgr.y.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38307.log.gz 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41435.log 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40810.log 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39715.log 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38934.log.gz 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.41435.log.gz 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39715.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40810.log.gz 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38504.log 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39715.log.gz 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41200.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38504.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2024-03-29T10:50:01.785 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38504.log.gz 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38346.log 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.38346.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40966.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38346.log.gz 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gz 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34538.log 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39129.log 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34538.log.gz 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39129.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39129.log.gz 2024-03-29T10:50:01.786 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34932.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39168.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.34932.log: /var/log/ceph/ceph-client.admin.39597.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34932.log.gz 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41278.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39558.log: /var/log/ceph/ceph-client.admin.39168.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39168.log.gz 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39324.log: /var/log/ceph/ceph-client.admin.41278.log: 86.6% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41278.log.gz 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39793.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41239.log: /var/log/ceph/ceph-client.admin.39793.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2024-03-29T10:50:01.787 INFO:teuthology.orchestra.run.smithi094.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39793.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39090.log 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.40066.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39949.log: /var/log/ceph/ceph-client.admin.39090.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39090.log.gz 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40459.log 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39871.log 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41591.log: /var/log/ceph/ceph-client.admin.40459.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40459.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39285.log 2024-03-29T10:50:01.788 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:50:01.792 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.39871.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39871.log.gz 2024-03-29T10:50:01.792 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41669.log 2024-03-29T10:50:01.792 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:50:01.792 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.41669.log: /var/log/ceph/ceph-client.admin.39285.log: 87.3% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39285.log.gz 2024-03-29T10:50:01.792 INFO:teuthology.orchestra.run.smithi094.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41669.log.gz 2024-03-29T10:50:01.796 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:50:01.796 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:50:01.796 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:50:01.796 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34214.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34294.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34214.log: /var/log/ceph/ceph-client.admin.34294.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.7.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34214.log.gz 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.34294.log.gz --verbose 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr: -- /var/log/ceph/ceph-client.admin.34374.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34414.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-mgr.x.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34374.log.gz 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34414.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34454.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34414.log.gz 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34494.log 2024-03-29T10:50:01.797 INFO:teuthology.orchestra.run.smithi173.stderr: 87.2%/var/log/ceph/ceph-client.admin.34454.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:50:01.798 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34254.log 2024-03-29T10:50:01.798 INFO:teuthology.orchestra.run.smithi173.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34454.log.gz 2024-03-29T10:50:01.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34334.log 2024-03-29T10:50:01.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34254.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34494.log.gz 2024-03-29T10:50:01.798 INFO:teuthology.orchestra.run.smithi173.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34254.log.gz 2024-03-29T10:50:01.798 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-client.admin.34334.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34334.log.gz 2024-03-29T10:50:01.798 INFO:teuthology.orchestra.run.smithi173.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:50:01.827 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:50:12.808 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:50:12.955 INFO:teuthology.orchestra.run.smithi094.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:50:13.417 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:50:13.730 INFO:teuthology.orchestra.run.smithi094.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:50:16.839 INFO:teuthology.orchestra.run.smithi094.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:50:17.201 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:50:17.219 INFO:teuthology.orchestra.run.smithi173.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:50:17.962 INFO:teuthology.orchestra.run.smithi094.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:50:50.466 INFO:teuthology.orchestra.run.smithi094.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:50:54.929 INFO:teuthology.orchestra.run.smithi094.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:50:54.931 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-03-29T10:50:54.931 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m53.209s 2024-03-29T10:50:54.931 INFO:teuthology.orchestra.run.smithi094.stderr:user 2m30.559s 2024-03-29T10:50:54.932 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m5.386s 2024-03-29T10:51:06.229 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:51:06.230 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-29T10:51:06.230 INFO:teuthology.orchestra.run.smithi173.stderr:real 1m4.471s 2024-03-29T10:51:06.231 INFO:teuthology.orchestra.run.smithi173.stderr:user 1m53.472s 2024-03-29T10:51:06.231 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m4.225s 2024-03-29T10:51:06.231 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:51:06.231 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/remote/smithi094/log 2024-03-29T10:51:06.232 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:51:29.999 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/remote/smithi173/log 2024-03-29T10:51:30.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:51:47.049 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:51:47.252 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c8453bf189e7494ca78f09a4cdeb61e9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:51:47.255 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:51:47.264 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:51:47.272 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:51:47.273 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:51:47.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:51:47.366 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:51:47.375 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:51:47.375 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:51:47.379 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:51:47.392 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:51:47.393 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:51:47.397 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:51:47.397 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-29T10:51:47.397 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +414us[ +392us] +/- 70ms 2024-03-29T10:51:47.398 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -196us[ -218us] +/- 50ms 2024-03-29T10:51:47.398 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 -216us[ -238us] +/- 80ms 2024-03-29T10:51:47.398 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:47.399 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:51:47.399 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-03-29T10:51:47.400 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +443us[ +443us] +/- 70ms 2024-03-29T10:51:47.400 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -179us[ -196us] +/- 49ms 2024-03-29T10:51:47.400 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 -213us[ -213us] +/- 80ms 2024-03-29T10:51:47.400 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:47.400 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:51:47.409 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:51:47.409 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:51:47.419 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:51:47.509 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:51:47.574 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:51:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:51:47.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:51:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:51:47.637 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:51:47.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:51:47.655 INFO:teuthology.task.internal:Duration was 1230.221488 seconds 2024-03-29T10:51:47.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:51:47.665 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:51:47.665 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:51:47.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:51:47.711 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:51:47.720 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:51:48.127 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:51:48.127 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-03-29T10:51:48.127 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:51:48.148 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-03-29T10:51:48.148 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:51:48.172 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:51:48.173 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:51:48.191 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:51:48.241 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:51:48.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:51:48.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:51:48.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:51:48.393 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:51:48.393 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:51:48.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:51:48.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:51:48.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:51:48.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:51:48.485 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-03-29T10:51:48.512 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-03-29T10:51:48.534 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:51:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:51:48.570 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:51:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:51:48.588 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:51:48.599 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:51:48.601 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/remote/smithi094 2024-03-29T10:51:48.601 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:51:48.647 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646/remote/smithi173 2024-03-29T10:51:48.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:51:48.682 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:51:48.682 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:51:48.684 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:51:48.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:51:48.742 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:51:48.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:51:48.751 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:51:48.751 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:51:48.754 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:51:48.766 INFO:teuthology.orchestra.run.smithi094.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:51 /home/ubuntu/cephtest 2024-03-29T10:51:48.785 INFO:teuthology.orchestra.run.smithi173.stdout: 265610 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:51 /home/ubuntu/cephtest 2024-03-29T10:51:48.786 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:51:48.798 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:51:48.819 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:51:48.846 DEBUG:teuthology.nuke:shortname: smithi094 2024-03-29T10:51:48.846 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:51:48.870 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:23:03.018362', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBET4uWJ1dRCY8i6GpiJNqX+aKDBuXu7e+8lWovKubi62FhnlCX+Wyv2xAdWjc9TNzP2I0g6FSbsqwBbCUYvKKv4='} 2024-03-29T10:51:48.875 DEBUG:teuthology.nuke:shortname: smithi173 2024-03-29T10:51:48.876 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:51:48.897 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622646', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:23:03.019772', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA0TujK83NAzIr2yYFWZMtxTPs5Qz3YLnVMBZnma4ddeq0wqf85ZCyNNfWNT6kIfQwZeb7neKme59t5iHy5j8Oo='} 2024-03-29T10:51:48.927 INFO:teuthology.orchestra.console.smithi094:Power off 2024-03-29T10:51:48.928 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:51:48.953 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2024-03-29T10:51:48.953 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:51:48.969 INFO:teuthology.orchestra.console.smithi173:Power off 2024-03-29T10:51:48.969 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:51:48.979 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-29T10:51:48.993 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-03-29T10:51:48.993 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:51:49.024 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-29T10:51:52.980 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:51:53.026 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:51:53.092 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2024-03-29T10:51:53.138 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-29T10:51:57.095 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:51:57.138 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:51:57.209 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2024-03-29T10:51:57.251 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-03-29T10:51:57.310 INFO:teuthology.orchestra.console.smithi094:Power off completed 2024-03-29T10:51:57.353 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-03-29T10:51:57.458 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1230.2214875221252 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c8453bf189e7494ca78f09a4cdeb61e9 status: fail success: false 2024-03-29T10:51:57.459 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:51:57.543 INFO:teuthology.run:FAIL